Repository: nvaccess/nvda

Issue:
No item names for some listviews (64-bit)
### Steps to reproduce:
Example app: Tortoise SVN

- invoke a context command such as SVN Update on a local repo.
- A window appears with the progress made so far.
- Tab to the list and try to navigate.
- one or multiple  error sounds are issued.

### Expected behavior:
The name of the focused item should be announced.

### Actual behavior:
Only the state (e.g. selected) is announced

### System configuration:
NVDA version next-15028,2754a2c8
Installed/Source/portable

Windows 10 (x64), Version 1803 (OS Build 17133.73)

Name and version of other software in use when reproducing the issue:
Tortoise SVN 1.9.7.27907 x64
### Other questions:

Does the issue still occur after restarting your PC?

Yes

Have you tried any other versions of NVDA?

Not relevant



Discussion:
I am also very uncomfortable with this problem. This issue can be reproduced in both 2018.1 and master, next versions.
## Background
The list in question is defined in:

> NVDAObjects.IAccessible.sysListView32

The problematic class is:

> ListItemWithoutColumnSupport(IAccessible)

and the method involved:

> _getColumnContentRaw(self, index)

### General Code Flow:

- virtual memory is allocated in the 64-bit  process
- One part is for the receiving buffer
- the other part is for the LVITEM, a special structure that is sent with the message
- watchdog.cancellableSendMessage is called with msg = LVM_GETITEMTEXTW and the structure mentioned above
- the listview item now holds a pointer to the item name
- NVDA makes an attempt to read the buffer
- fail

### Debug Result
Note that this is made with NVDA in source mode and two helper functions:

- a Repr class added to the LVITEM structure to show its members at each step
- Printout of the arguments that are passed to "winKernel.readProcessMemory"

The command (after selecting an item in the list view) is:

	>>> focus._getColumnContentRaw( <index> )

#### 1. The structure written to VM:
		mask 513 # set text and columns
		iItem 0
		iSubItem 0
		state 0
		stateMask 0
		pszText 57671680 # desired address of the receiving buffer
		cchTextMax 512 # length
		iImage 0
		lParam 0
		iIndent 0
		iGroupID 0
		cColumns 0
		puColumns 0
		piColFmt 0
		iGroup 0

#### 2. the arguments passed to "ReadProcessMemory":

		2372 <type 'int'> # Process Handle
		57606144 <type 'int'> # Base Address
		<cparam 'P' (06AE9DB8)> <type 'CArgObject'> # the structure
		80 <type 'int'> # size in bytes
		None <type 'NoneType'>

#### 3. The returned structure 

		mask 1 # Text
		iItem 0
		iSubItem 0
		state 0
		stateMask 0
		pszText 669217843296 # didn't use our address!!
		cchTextMax 512 # length was accepted though
		iImage 0
		lParam 0
		iIndent 0
		iGroupID 0
		cColumns 0
		puColumns 0
		piColFmt 0
		iGroup 0

#### 4. The arguments for reading the buffer:

		2372 <type 'int'>
		669217843296 <type 'long'> # Note that int is now long which is actually c_ulonglong
		<ctypes.c_wchar_Array_1 object at 0x06CDC490> <class 'ctypes.c_wchar_Array_1'>
		2 <type 'int'>
		None <type 'NoneType'>

#### 5. The resulting error:

		Traceback (most recent call last):
		  File "<console>", line 1, in <module>
		  File "C:\nvda\source\NVDAObjects\IAccessible\sysListView32.py", line 341, in _getColumnContentRaw
			winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)
		  File "C:\nvda\source\winKernel.py", line 180, in readProcessMemory
			return kernel32.ReadProcessMemory(*args)
		ArgumentError: argument 2: <type 'exceptions.OverflowError'>: long int too long to convert


### Conclusion:
- The listview control doesn't accept our pointer to the buffer and replaces it with a 64-bit pointer.
- From our 32-bit process, it is impossible to read the buffer.

Note that all other functions in this class that use the similar procedure work fine.
So can the column headers be read without problems.

There is nothing mentioned on MSDN with regard to the LVM_GETITEMTEXTW message.

However, this can be found under  the similar LVM_GETITEMW "Remarks" section:

> If the LVIF_TEXT flag is set in the mask member of the LVITEM structure, the pszText member must point to a valid buffer and the cchTextMax member must be set to the number of characters in that buffer.  
> Applications should not assume that the text will necessarily be placed in the specified buffer.  
> The control may instead change the pszText member of the structure to point to the new text, rather than place it in the buffer.

<https://msdn.microsoft.com/en-us/library/windows/desktop/bb774953(v=vs.85).aspx>
I just closed #7874 in favour of this issue. @Robert-J-H: Thanks for providing a technical background.
Hi,

Because we did have issues with this particular structure, and since we now have more detailed information, I recommend keeping this one in favor of others. Thanks. CC @JCSTeh, @MichaelDCurran
Thanks @leonardder

I should have investigated a bit deeper and commented on the open issue.
However, the problem is more general and affects probably all 64-bit applications that use this class (but I've only encountered Tortoise).
Note that there is another bug hidden in the method.
If you look at the arguments under 4.0, there is the integer "2" right above "None".
This is the buffer length, however, this can't be since we are using 512 characters, no matter what.
Notice how the variable len (bad name by the way) is used to create the buffer within the relevant code snippet:

```
len = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETITEMTEXTW, (self.IAccessibleChildID-1), internalItem)
				if len:
					winKernel.readProcessMemory(processHandle,internalItem,byref(item),sizeof(self.LVITEM),None)
					buffer=create_unicode_buffer(len)
					winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)

```

For the buffer creation "item.cchTextMax" should be used instead.
LVM_GETITEMTEXT is documented to return the length of the text, which is why we do (and it's appropriate for us to) use it as the length of the buffer. The issue with 64 bit apps is that NVDA, as a 32 bit app, can't accept remote 64 bit memory addresses. There's another issue open somewhere to deal with this using our own in-process memory reading/writing helpers.

@jcsteh
Thanks James,
I've just realized that len is the amount of characters.
The _getHeaderRaw method does it curiously in the other way:
```
	res = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETCOLUMNW, index, internalColumn)
	if res:
		winKernel.readProcessMemory(processHandle,internalColumn,byref(column),sizeof(self.LVCOLUMN),None)
		buffer=create_unicode_buffer(column.cchTextMax)
		winKernel.readProcessMemory(processHandle,column.pszText,buffer,sizeof(buffer),None)```

I agree with you, an external Helper function is probably needed.
I first contemplated replacing the text pointer with a callback (-1) but this would only move the problem to another place.
[Using text callbacks in ListView Controls](https://www.codeproject.com/Articles/70/Using-text-callbacks-in-ListView-Controls)
I've created pull request #8190 that addresses this issue, although not at a IPC level.
I think that it would be embarrassing if we recommended Tortoise SVN for translators without proper accessibility for the program.
I hope that some of you have other programs that exhibit the same bug and can be tested with the workaround.
I'm using NVDA version alpha-15693,b46b83ea and now the reading of the log list in TortoiseSVN is broken again. Was this commit one of the ones lost during the change of the testing/release process a few weeks ago?
Hi, yes. Thanks.
If I read the two above comments correctly, shouldn't the patch for PR #8190 be merged again or am I missing the point?
I don't understand why it was lost and never merged again...
Sure, it could have been merged some two years ago.
Seems that NV Access wants another way but I'm not sure if it will
work so easily within the C++ code.

On 13/05/2020, ruifontes <notifications@github.com> wrote:
> I don't understand why it was lost and never merged again...
>
> --
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub:
> https://github.com/nvaccess/nvda/issues/8175#issuecomment-627666290

Bumping this because it's also been identified as a show-stopper issue for Sibiac (an NVDA addon that provides access to DAW plugins in context of music production). Any news on that merge?
cc @michaelDCurran, @feerrenrut 
In which way does it tangle Sibiac?
I use Azlo's addon  daily, although be it with Python 2.
For now, the author could just overwrite the SysListView32 class with
this version and work from there.


Hey @Robert-J-H , my admittedly layman understanding of how this relates to Sibiac is that with some 64-bit plugins, this issue affects NVDA's handling of the FX chain list view, which is where Sibiac recognizes the plugin that's loaded and determines which overlay to use. CC-ing @Azslow3, he's better placed to explain what's going on.
Hm, I see some problems in the FX Chain list box. It often stops to
read the actual VST Plug-in (Same in the browser) and NVDA reports
some out of memory error.
Which is hardly the case with 32 GB.
Anyway, Reapi has a function to return the current dll associated with
the focused plug-in. This makes sense as plug-in instances can have
any name if the user wishes so.
Apart from the memory leak that I've mentioned above, the lists work
nicely in Reaper, including name and number of entries. In contrast,
the lack of those would mark the issue addressed in this thread.
The MRO for a list item is (ignore the column review part, it's
created by an addon):
Python class mro: (<class
'NVDAObjects.Dynamic_ColumnsReview32ListItemIAccessible'>, <class
'globalPlugins.columnsReview.ColumnsReview32'>, <class
'globalPlugins.columnsReview.ColumnsReview'>, <class
'NVDAObjects.IAccessible.sysListView32.ListItem'>, <class
'NVDAObjects.behaviors.RowWithFakeNavigation'>, <class
'NVDAObjects.behaviors.RowWithoutCellObjects'>, <class
'NVDAObjects.IAccessible.sysListView32.ListItemWithoutColumnSupport'>,
<class 'NVDAObjects.IAccessible.IAccessible'>, <class
'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class
'documentBase.TextContainerObject'>, <class
'baseObject.ScriptableObject'>, <class
'baseObject.AutoPropertyObject'>, <type 'object'>)


I'm confused. The FX Chain list reads just fine for me in 64 bit REAPER with NVDA...?
There is no strait way to reproduce the problem in REAPER. So far I cood hit it with Toontrack EZ Drummer and XLN Addictive Drums VSTi only, in relatively busy project. These plug-ins fill all lower 32bit address space during GUI loading. Can be somehow JUCE related, so probably affect many plug-ins.

Synthetic test is in http://www.azslow.com/index.php?action=downloads;sa=downfile&id=77
atest.exe fills all 32bit space. Both continuously monitor REAPER, when it is running. That way I have checked the reason in REAPER is the same.

If I understand that right, IAccessible works using the same approach. So many things can break once 32bit addresses are reserved in the target. That is not a bug on 64bit application side, modern programs grow in size, VirtualAlloc is not unusual call inside applications and consumes lower address space by default. Not good for 32bit accessibility applications.
[Here](https://ci.appveyor.com/api/buildjobs/efjg439icangaju4/artifacts/output%2Fnvda_snapshot_try-listView-21157%2Cb6ea4911.exe) is a try build that hopefully fixes this issue.
Good job! Thanks! It is working on my end.
Let's hope it is merged soon, I was expecting it.

Hey @leonardder, just tested the try build above in Reaper X64, and I'm still getting broken list views in the FX window once memory usage of the project exceeds 4GB. Is there anything I can provide to help diagnose? NVDA log etc?
@Scott, yes do so. Is the error the same as in the normal built?

@ScottChesworth A log would be pretty helpful. However, I'm pretty sure this error has a different background and requires more porting to C++.
Hi @ScottChesworth could you please provide an NVDA log for this issue, or advise if it still occurs?
@feerrenrut, yep, still present in the public release of NVDA 2020.4. In the software I use most often (Reaper X64), any plug-in included in the "List1" control of Reaper's FX window in a project that uses over 4GB of RAM seems to be susceptible on my machine. It doesn't happen every time those conditions are met here, but it does happen often. I seem to be having a lucky day today, will pop a log in here next time I experience it. Thanks for the nudge, I should've done that ages ago.
Confirmed here. The list in the FX Browser is affected as well.


I've been seeing this in REAPER in all ListViews (FX chain list, FX add list, Actions list) with large projects too. I didn't grab a log when it was happening, though. :(
It's primarily in large projects when it happens here. Annoyingly it always seems to crop up when I'm against the clock. Will try harder to remember that I need to grab a log next time it occurs.
@AZSlow3's synthetic test case in https://github.com/nvaccess/nvda/issues/8175#issuecomment-638705839 makes it ridiculously easy to reproduce this. Thanks.

Log:

```
IO - inputCore.InputManager.executeGesture (21:02:55.303) - winInputHook (16888):
Input: kb(desktop):downArrow
ERROR - eventHandler.executeEvent (21:02:55.337) - MainThread (23152):
error executing event: gainFocus on <NVDAObjects.IAccessible.sysListView32.ListItem object at 0x06F87A90> with extra args of {}
Traceback (most recent call last):
  File "eventHandler.pyc", line 284, in executeEvent
  File "eventHandler.pyc", line 98, in __init__
  File "eventHandler.pyc", line 107, in next
  File "C:\Users\jamie\AppData\Roaming\nvda\addons\remote\globalPlugins\remoteClient\__init__.py", line 458, in event_gainFocus
    nextHandler()
  File "eventHandler.pyc", line 107, in next
  File "NVDAObjects\__init__.pyc", line 1173, in event_gainFocus
  File "NVDAObjects\behaviors.pyc", line 581, in reportFocus
  File "NVDAObjects\__init__.pyc", line 1028, in reportFocus
  File "speech\speech.pyc", line 555, in speakObject
  File "speech\speech.pyc", line 598, in getObjectSpeech
  File "speech\speech.pyc", line 450, in getObjectPropertiesSpeech
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 451, in _get_name
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 357, in _getColumnLocation
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 220, in _get__columnOrderArray
  File "winKernel.pyc", line 278, in virtualAllocEx
OSError: [WinError 8] Not enough memory resources are available to process this command.
```

Fixing this will require moving `_get__columnOrderArray` to C++, as well as `_getColumnLocationRaw` and `_getColumnHeaderRaw`. You don't see exceptions here for the latter two, but that's only because the code never hits that point. I verified (with the Python console) that those throw the same exception too. There's also `_getColumnImageIDRaw`, but that only gets used by Outlook Express, so we can probably ignore that.

`_get__columnOrderArray` is going to be interesting because it needs to return an array of ints and raw RPC isn't great at handling that kind of thing. It's likely that a BSTR hack similar to `displayModel_getWindowTextInRect` will be necessary where we shove ints into a BSTR, casting them back and forth from wchars.
To further clarify, the difference between this issue and the earlier one is that with the earlier (now fixed) one, we can successfully allocate 32 bit memory in the remote process, but the control decides to give us back memory beyond 32 bit and we fail to read it. In this case, we can't allocate any 32 bit memory in the remote process at all because it's been exhausted.
I can confirm that my current workaround (own 64bit proxy) overwrites columnOrderArray for the List and getColumnLocationRaw for ListItem. That way FX List in REAPER works correctly (that list has no Header).

Resolving patch:
diff --git a/source/NVDAObjects/IAccessible/sysListView32.py b/source/NVDAObjects/IAccessible/sysListView32.py
--- a/source/NVDAObjects/IAccessible/sysListView32.py
+++ b/source/NVDAObjects/IAccessible/sysListView32.py
@@ -23,6 +23,7 @@
 import config
 from locationHelper import RectLTRB
 from logHandler import log
+from typing import Optional
 
 #Window messages
 LVM_FIRST=0x1000
@@ -228,13 +229,31 @@ def _get_columnCount(self):
 			return 1
 		return count
 
-	def _get__columnOrderArray(self):
-		coa=(c_int *self.columnCount)()
+	def _getColumnOrderArrayRaw(self, columnCount: int) -> Optional[ctypes.Array]:
+		columnOrderArray = (ctypes.c_int * columnCount)()
+		res = watchdog.cancellableExecute(
+			NVDAHelper.localLib.nvdaInProcUtils_sysListView32_getColumnOrderArray,
+			self.appModule.helperLocalBindingHandle,
+			self.windowHandle,
+			columnCount,
+			columnOrderArray
+		)
+		if res:
+			return None
+		return columnOrderArray
+
+	def _getColumnOrderArrayRawOutProc(self, columnCount: int) -> Optional[ctypes.Array]:
+		coa = (ctypes.c_int * columnCount)()
 		processHandle=self.processHandle
 		internalCoa=winKernel.virtualAllocEx(processHandle,None,sizeof(coa),winKernel.MEM_COMMIT,winKernel.PAGE_READWRITE)
 		try:
 			winKernel.writeProcessMemory(processHandle,internalCoa,byref(coa),sizeof(coa),None)
-			res = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETCOLUMNORDERARRAY, self.columnCount, internalCoa)
+			res = watchdog.cancellableSendMessage(
+				self.windowHandle,
+				LVM_GETCOLUMNORDERARRAY,
+				columnCount,
+				internalCoa
+			)
 			if res:
 				winKernel.readProcessMemory(processHandle,internalCoa,byref(coa),sizeof(coa),None)
 			else:
@@ -243,6 +262,12 @@ def _get__columnOrderArray(self):
 			winKernel.virtualFreeEx(processHandle,internalCoa,0,winKernel.MEM_RELEASE)
 		return coa
 
+	def _get__columnOrderArray(self) -> Optional[ctypes.Array]:
+		if not self.appModule.helperLocalBindingHandle:
+			return self._getColumnOrderArrayRawOutProc(self.columnCount)
+		return self._getColumnOrderArrayRaw(self.columnCount)
+
+
 class GroupingItem(Window):
 
 	def __init__(self,windowHandle=None,parentNVDAObject=None,groupInfo=None):
@@ -321,7 +346,22 @@ def event_stateChange(self):
 
 class ListItem(RowWithFakeNavigation, RowWithoutCellObjects, ListItemWithoutColumnSupport):
 
-	def _getColumnLocationRaw(self,index):
+	def _getColumnLocationRaw(self, index: int) -> ctypes.wintypes.RECT:
+		item = self.IAccessibleChildID - 1
+		subItem = index
+		rect = ctypes.wintypes.RECT()
+		if watchdog.cancellableExecute(
+			NVDAHelper.localLib.nvdaInProcUtils_sysListView32_getColumnLocation,
+			self.appModule.helperLocalBindingHandle,
+			self.windowHandle,
+			item,
+			subItem,
+			ctypes.byref(rect)
+		) != 0:
+			return None
+		return rect
+
+	def _getColumnLocationRawOutProc(self, index: int) -> ctypes.wintypes.RECT:
 		processHandle=self.processHandle
 		# LVM_GETSUBITEMRECT requires a pointer to a RECT structure that will receive the subitem bounding rectangle information.
 		localRect=RECT(
@@ -353,24 +393,31 @@ def _getColumnLocationRaw(self,index):
 		if res == 0:
 			log.debugWarning(f"LVM_GETSUBITEMRECT failed for index {index} in list")
 			return None
+		return localRect
+
+	def _getColumnLocation(self, column: int) -> Optional[RectLTRB]:
+		index = self.parent._columnOrderArray[column - 1]
+		if not self.appModule.helperLocalBindingHandle:
+			rect = self._getColumnLocationRawOutProc(index)
+		else:
+			rect = self._getColumnLocationRaw(index)
+		if rect is None:
+			return None
 		# #8268: this might be a malformed rectangle
 		# (i.e. with a left coordinate that is greater than the right coordinate).
 		# This happens in Becky! Internet Mail,
 		# as well in applications that expose zero width columns.
-		left = localRect.left
-		top = localRect.top
-		right = localRect.right
-		bottom = localRect.bottom
+		left = rect.left
+		top = rect.top
+		right = rect.right
+		bottom = rect.bottom
 		if left > right:
 			left = right
 		if top > bottom:
 			top = bottom
 		return RectLTRB(left, top, right, bottom).toScreen(self.windowHandle).toLTWH()
 
-	def _getColumnLocation(self,column):
-		return self._getColumnLocationRaw(self.parent._columnOrderArray[column - 1])
-
-	def _getColumnContentRaw(self, index):
+	def _getColumnContentRaw(self, index: int) -> Optional[str]:
 		item = self.IAccessibleChildID - 1
 		subItem = index
 		text = AutoFreeBSTR()
@@ -385,7 +432,7 @@ def _getColumnContentRaw(self, index):
 			return None
 		return text.value
 
-	def _getColumnContentRawOutProc(self, index):
+	def _getColumnContentRawOutProc(self, index: int) -> Optional[str]:
 		buffer=None
 		processHandle=self.processHandle
 		internalItem=winKernel.virtualAllocEx(processHandle,None,sizeof(self.LVITEM),winKernel.MEM_COMMIT,winKernel.PAGE_READWRITE)
@@ -405,7 +452,7 @@ def _getColumnContentRawOutProc(self, index):
 			winKernel.virtualFreeEx(processHandle,internalItem,0,winKernel.MEM_RELEASE)
 		return buffer.value if buffer else None
 
-	def _getColumnContent(self, column):
+	def _getColumnContent(self, column: int) -> Optional[str]:
 		index = self.parent._columnOrderArray[column - 1]
 		if not self.appModule.helperLocalBindingHandle:
 			return self._getColumnContentRawOutProc(index)
@@ -428,7 +475,7 @@ def _getColumnImageIDRaw(self, index):
 	def _getColumnImageID(self, column):
 		return self._getColumnImageIDRaw(self.parent._columnOrderArray[column - 1])
 
-	def _getColumnHeaderRaw(self,index):
+	def _getColumnHeaderRawOutProc(self, index: int) -> Optional[str]:
 		buffer=None
 		processHandle=self.processHandle
 		internalColumn=winKernel.virtualAllocEx(processHandle,None,sizeof(self.LVCOLUMN),winKernel.MEM_COMMIT,winKernel.PAGE_READWRITE)
@@ -448,8 +495,24 @@ def _getColumnHeaderRaw(self,index):
 			winKernel.virtualFreeEx(processHandle,internalColumn,0,winKernel.MEM_RELEASE)
 		return buffer.value if buffer else None
 
-	def _getColumnHeader(self, column):
-		return self._getColumnHeaderRaw(self.parent._columnOrderArray[column - 1])
+	def _getColumnHeaderRaw(self, index: int) -> Optional[str]:
+		subItem = index
+		text = AutoFreeBSTR()
+		if watchdog.cancellableExecute(
+			NVDAHelper.localLib.nvdaInProcUtils_sysListView32_getColumnHeader,
+			self.appModule.helperLocalBindingHandle,
+			self.windowHandle,
+			subItem,
+			ctypes.byref(text)
+		) != 0:
+			return None
+		return text.value
+
+	def _getColumnHeader(self, column: int) -> Optional[str]:
+		index = self.parent._columnOrderArray[column - 1]
+		if not self.appModule.helperLocalBindingHandle:
+			return self._getColumnHeaderRawOutProc(index)
+		return self._getColumnHeaderRaw(index)
 
 	def _get_name(self):
 		parent = self.parent


Test patch: