Repository: nvaccess/nvda

Issue:
Cancelable speech. NVDA freezes with this character
### Steps to reproduce:
1. With NVDA running, enable cancelable speech in the advanced settings
2. Open this website and navigate to the input edit field under html fonts
https://www.htmlsymbols.xyz/unicode/U+000B
3. Alternatively, insert the character with the unicode value U+000B in any of the editors (i.e. MS Word)
4. Navigate with arrow keys on the character, no matter in focus or browse mode

### Actual behavior:
NVDA freezes and needs to be restarted. Following error is shown in the nvda_old.log:

```
IO - inputCore.InputManager.executeGesture (18:07:44.957) - winInputHook (11300):
Input: kb(laptop):control+rightArrow
DEBUG - editableText.EditableText._hasCaretMoved (18:07:45.003) - MainThread (3272):
Caret move detected using event. Elapsed: 0 ms
IO - inputCore.logTimeSinceInput (18:07:45.046) - MainThread (3272):
0.089 sec since input
IO - speech.speak (18:07:45.046) - MainThread (3272):
Speaking ['Tab', EndUtteranceCommand()]
IO - inputCore.InputManager.executeGesture (18:07:45.536) - winInputHook (11300):
Input: kb(laptop):rightArrow
DEBUG - editableText.EditableText._hasCaretMoved (18:07:45.575) - MainThread (3272):
Caret move detected using event. Elapsed: 0 ms
IO - inputCore.logTimeSinceInput (18:07:45.615) - MainThread (3272):
0.078 sec since input
IO - speech.speak (18:07:45.615) - MainThread (3272):
Speaking [CharacterModeCommand(True), '\x0b', EndUtteranceCommand()]
ERROR - speech.manager.SpeechManager._getUtteranceIndex (18:07:45.615) - MainThread (3272):
Expected last item to be an indexCommand.
ERROR - speech.manager.SpeechManager._checkForCancellations (18:07:45.639) - MainThread (3272):
no utterance index, cant save cancellable commands

DEBUGWARNING - watchdog._watcher (18:07:50.150) - watchdog (8832):
Trying to recover from freeze. Listing stacks for Python threads:
Python stack for thread 8832 (watchdog):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "threading.pyc", line 870, in run
  File "watchdog.pyc", line 127, in _watcher
  File "watchdog.pyc", line 63, in getFormattedStacksForAllThreads

Python stack for thread 2304 (Bluetooth Audio background beeper thread):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "C:\Users\adria\AppData\Roaming\nvda\addons\bluetoothaudio\globalPlugins\bluetoothAudio.py", line 52, in run
    playerLocal.feed(buf.raw)
  File "nvwave.pyc", line 326, in feed
  File "nvwave.pyc", line 341, in _feedUnbuffered_handleErrors
  File "nvwave.pyc", line 369, in _feedUnbuffered
  File "nvwave.pyc", line 386, in sync
  File "winKernel.pyc", line 228, in waitForSingleObject

Python stack for thread 13296 (touchHandler.TouchHandler):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "touchHandler.pyc", line 241, in run

Python stack for thread 11300 (winInputHook):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "threading.pyc", line 870, in run
  File "winInputHook.pyc", line 79, in hookThreadFunc

Python stack for thread 12140 (_UIAHandler.UIAHandler.MTAThread):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "threading.pyc", line 870, in run
  File "_UIAHandler.pyc", line 310, in MTAThreadFunc
  File "queue.pyc", line 170, in get
  File "threading.pyc", line 296, in wait

Python stack for thread 6672 (Thread-3):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "C:\Users\adria\AppData\Roaming\nvda\addons\IBMTTS\synthDrivers\_ibmeci.py", line 291, in run
    func, args, kwargs = callbackQueue.get()
  File "queue.pyc", line 170, in get
  File "threading.pyc", line 296, in wait

Python stack for thread 6560 (Thread-2):
  File "threading.pyc", line 890, in _bootstrap
  File "threading.pyc", line 926, in _bootstrap_inner
  File "C:\Users\adria\AppData\Roaming\nvda\addons\IBMTTS\synthDrivers\_ibmeci.py", line 127, in run
    user32.GetMessageA(byref(msg), 0, 0, 0)

Python stack for thread 3272 (MainThread):
File "nvwave.pyc", line 516, in _getOutputDevices
```



### Expected behavior:
No error and NVDA is not freezing. When disabling speech cancelation, everything works as expected.


### System configuration
#### NVDA installed/portable/running from source:
Installed

#### NVDA version:
Version: alpha-21145,ac31a795

#### Windows version:
Windows 10 2004 update

#### Name and version of other software in use when reproducing the issue:
MS Word 365, Chrome 85, Firefox 81.

#### Other information about your system:

### Other questions
#### Does the issue still occur after restarting your computer?
yes

#### Have you tried any other versions of NVDA? If so, please report their behaviors.
no

#### If addons are disabled, is your problem still occuring?
yes

#### Did you try to run the COM registry fixing tool in NVDA menu / tools?
yes, no effect.


Discussion:
Note that this might occur with other line ending or line tabulation characters as well. This needs further testing.
cc: @feerrenrut, @michaelDCurran 
I can reproduce this. Here is a log fragment from arrowing onto the character with speech manager debug logging enabled.
```
IO - inputCore.InputManager.executeGesture (08:18:00.579) - winInputHook (24560):
Input: kb(laptop):rightArrow
DEBUG - speech.manager.SpeechManager.cancel (08:18:00.609) - MainThread (26568):
SpeechManager- Cancel
IO - inputCore.logTimeSinceInput (08:18:00.609) - MainThread (26568):
0.030 sec since input
IO - speech.speak (08:18:00.609) - MainThread (26568):
Speaking [CharacterModeCommand(True), '\x0b', EndUtteranceCommand()]
DEBUG - speech.manager.SpeechManager.speak (08:18:00.609) - MainThread (26568):
SpeechManager- speak (priority <SpeechPriority.NORMAL: 0>): [CharacterModeCommand(True), '', EndUtteranceCommand()]
DEBUG - speech.manager.SpeechManager._queueSpeechSequence (08:18:00.609) - MainThread (26568):
SpeechManager- Out Seq: [[CharacterModeCommand(True), '']]
DEBUG - speech.manager.SpeechManager._queueSpeechSequence (08:18:00.609) - MainThread (26568):
SpeechManager- Current priority: 0, queLen: 0
DEBUG - speech.manager.SpeechManager._getMostRecentlyCancelledUtterance (08:18:00.609) - MainThread (26568):
SpeechManager- Length of _cancelCommandsForUtteranceBeingSpokenBySynth: 0 Length of _indexesSpeaking: 0 
DEBUG - speech.manager.SpeechManager._doRemoveCancelledSpeechCommands (08:18:00.609) - MainThread (26568):
SpeechManager- Last index: None
DEBUG - speech.manager.SpeechManager.speak (08:18:00.609) - MainThread (26568):
SpeechManager- Will interrupt: False Will push: True | _indexesSpeaking: [] | _curPriQueue valid: False | _shouldPushWhenDoneSpeaking: False | _cancelledLastSpeechWithSynth False
DEBUG - speech.manager.SpeechManager.speak (08:18:00.609) - MainThread (26568):
SpeechManager- Pushing next speech
DEBUG - speech.manager.SpeechManager._pushNextSpeech (08:18:00.615) - MainThread (26568):
SpeechManager- pushNextSpeech - doneSpeaking: True
ERROR - speech.manager.SpeechManager._getUtteranceIndex (08:18:00.615) - MainThread (26568):
Expected last item to be an indexCommand.
ERROR - speech.manager.SpeechManager._checkForCancellations (08:18:00.655) - MainThread (26568):
no utterance index, cant save cancellable commands
```
Note the last two errors repeat for ever until NVDA is restarted.
I think the next step will need to be reproducing this as a speech manager unit test. 
Arrowing to ASCII character 31, \x1f, or 28, (\x1c) will cause this to happen as well. 

Resolving patch:
diff --git a/source/speech/manager.py b/source/speech/manager.py
--- a/source/speech/manager.py
+++ b/source/speech/manager.py
@@ -333,7 +333,7 @@ def ensureEndUtterance(seq: SpeechSequence):
 			else:
 				lastOutSeq = outSeqs[-1] if outSeqs else None
 			lastCommand = lastOutSeq[-1] if lastOutSeq else None
-			if not lastCommand or isinstance(lastCommand, (EndUtteranceCommand, ConfigProfileTriggerCommand)):
+			if lastCommand is None or isinstance(lastCommand, (EndUtteranceCommand, ConfigProfileTriggerCommand)):
 				# It doesn't make sense to start with or repeat EndUtteranceCommands.
 				# We also don't want an EndUtteranceCommand immediately after a ConfigProfileTriggerCommand.
 				return
@@ -458,13 +458,18 @@ def _buildNextUtterance(self):
 		# apply any parameters changed before the preemption.
 		params = self._curPriQueue.paramTracker.getChanged()
 		utterance.extend(params)
-		for seq in self._curPriQueue.pendingSequences:
+		lastSequenceIndexAddedToUtterance = None
+		for seqIndex, seq in enumerate(self._curPriQueue.pendingSequences):
 			if isinstance(seq[0], EndUtteranceCommand):
 				# The utterance ends here.
 				break
 			utterance.extend(seq)
+			lastSequenceIndexAddedToUtterance = seqIndex
 		# if any items are cancelled, cancel the whole utterance.
 		if utterance and not self._checkForCancellations(utterance):
+			log.error(f"Checking for cancellations failed, cancelling sequence: {utterance}")
+			# Avoid infinite recursion by removing the problematic sequences:
+			del self._curPriQueue.pendingSequences[:lastSequenceIndexAddedToUtterance + 1]
 			return self._buildNextUtterance()
 		return utterance
 
@@ -542,7 +547,7 @@ def _getMostRecentlyCancelledUtterance(self) -> Optional[_IndexT]:
 		)
 		for index in cancelledIndexes:
 			if (
-				not latestCancelledUtteranceIndex
+				latestCancelledUtteranceIndex is None
 				or self._isIndexABeforeIndexB(latestCancelledUtteranceIndex, index)
 			):
 				latestCancelledUtteranceIndex = index


Test patch:
diff --git a/tests/unit/test_speechManager/__init__.py b/tests/unit/test_speechManager/__init__.py
--- a/tests/unit/test_speechManager/__init__.py
+++ b/tests/unit/test_speechManager/__init__.py
@@ -1139,12 +1139,15 @@ def setUp(self):
 		config.conf['featureFlag']['cancelExpiredFocusSpeech'] = 1  # yes
 
 
-class Test_pr11651(unittest.TestCase):
+class RegressionTests(unittest.TestCase):
+	"""Tests to prevent regressions after issues are fixed.
+	"""
 
 	def test_redundantSequenceAfterEndUtterance(self):
 		"""
 		Tests that redundant param change and index commands are not emitted as an extra utterance
 		when the preceeding utterance contained param change commands and an EndUtterance command.
+		See PR #11651
 		E.g. speaking a character.
 		"""
 		smi = SpeechManagerInteractions(self)
@@ -1168,3 +1171,28 @@ def test_redundantSequenceAfterEndUtterance(self):
 			smi.indexReached(1)
 			smi.doneSpeaking()
 			smi.pumpAll()
+
+	def test_nonSpokenCharacter(self):
+		"""Test for fix to GH#11752 - NVDA Freeze with unicode value U+000B
+		Actually, the speech manager receives an empty string for the character U+000B. NVDA
+		does not have a mapping for this character.
+		It is questionable whether we should send anything to the synth when there is no content, however
+		what constitutes 'content' is currently not easy to define.
+		"""
+		smi = SpeechManagerInteractions(self)
+
+		speechSequence = [
+			CharacterModeCommand(True),
+			'',
+			smi.create_EndUtteranceCommand(expectedToBecomeIndex=1)
+		]
+		with smi.expectation():
+			seqIndexes = smi.speak(speechSequence)
+			smi.expect_synthSpeak(seqIndexes)
+
+
+class RegressionTests_withCancellableSpeechEnabled(RegressionTests):
+	"""Note, while cancellable speech is configurable test with and without it enabled."""
+	def setUp(self):
+		super().setUp()
+		config.conf['featureFlag']['cancelExpiredFocusSpeech'] = 1  # yes