Repository: nvaccess/nvda

Issue:
NVDA's reporting of the remaining time is broken in Foobar2000
<!-- Please read the text in this edit field before filling it in.
Please thoroughly read NVDA's wiki article on how to fill in this template, including how to provide the required files.
Issues may be closed if the required information is not present.
https://github.com/nvaccess/nvda/wiki/Github-issue-template-explanation-and-examples
Please also note that the NVDA project has a Citizen and Contributor Code of Conduct which can be found at https://github.com/nvaccess/nvda/blob/master/CODE_OF_CONDUCT.md. NV Access expects that all contributors and other community members read and abide by the rules set out in this document while participating or contributing to this project. This includes creating or commenting on issues and pull requests. 
-->
Control+SHIFT+R reports the remaining time in Foobar2000, but now it doesn't work and NVDA remains silent.
### Steps to reproduce:
1. Run Foobar2000 and play a track.
2. Press Control+SHIFT+R.
### Actual behavior:
NVDA remains silent. Note that Control+SHIFT+E and Control+SHIFT+T still work.
### Expected behavior:
NVDA should report the remaining time.
### System configuration
#### NVDA installed/portable/running from source:
Installed
#### NVDA version:
NVDA Alpha 21882, but many alphas before it also exhibit the issue.
#### Windows version:
Win 10 Pro 64-Bit, Version 20H2 (OS Build 19042.867)
#### Name and version of other software in use when reproducing the issue:
foobar2000 version 1.6.2.0
#### Other information about your system:
I've even reset all Foobar 2000 settings, but the issue persists.
### Other questions
#### Does the issue still occur after restarting your computer?
Yes
#### Have you tried any other versions of NVDA? If so, please report their behaviors.

#### If add-ons are disabled, is your problem still occurring?
Yes
#### Did you try to run the COM registry fixing tool in NVDA menu / tools?
No


Discussion:
I've just yesterday noticed this issue on my end as well, but wanted to investigate if it was my Foobar. Thanks @amirsol81 for investigating, I can reproduce it here as well.
Here is a log demonstrating the issue:
[Foobar remaining time bug.log](https://github.com/nvaccess/nvda/files/6167680/Foobar.remaining.time.bug.log)

Hi, based on the log, it appears this is something to do with wxPython (again…) + Python 3.8. Thanks.


Any updates  on  this one? It can still be reproduced using alpha-22050,83738d7d

Resolving patch:
diff --git a/source/core.py b/source/core.py
--- a/source/core.py
+++ b/source/core.py
@@ -449,6 +449,9 @@ def handlePowerStatusChange(self):
 			locale.Init(wxLang.Language)
 		except:
 			log.error("Failed to initialize wx locale",exc_info=True)
+		finally:
+			# Revert wx's changes to the python locale
+			languageHandler.setLocale(languageHandler.curLang)
 
 	log.debug("Initializing garbageHandler")
 	garbageHandler.initialize()
diff --git a/source/languageHandler.py b/source/languageHandler.py
--- a/source/languageHandler.py
+++ b/source/languageHandler.py
@@ -13,6 +13,7 @@
 import locale
 import gettext
 import globalVars
+from logHandler import log
 
 #a few Windows locale constants
 LOCALE_SLANGUAGE=0x2
@@ -146,37 +147,119 @@ def getWindowsLanguage():
 			localeName="en"
 	return localeName
 
-def setLanguage(lang):
+
+def setLanguage(lang: str) -> None:
+	'''
+	Sets the following using `lang` such as "en", "ru_RU", or "es-ES". Use "Windows" to use the system locale
+	 - the windows locale for the thread (fallback to system locale)
+	 - the translation service (fallback to English)
+	 - languageHandler.curLang (match the translation service)
+	 - the python locale for the thread (match the translation service, fallback to system default)
+	'''
 	global curLang
-	try:
-		if lang=="Windows":
-			localeName=getWindowsLanguage()
-			trans=gettext.translation('nvda',localedir='locale',languages=[localeName])
-			curLang=localeName
-		else:
-			trans=gettext.translation("nvda", localedir="locale", languages=[lang])
-			curLang=lang
-			localeChanged=False
-			#Try setting Python's locale to lang
-			try:
-				locale.setlocale(locale.LC_ALL,lang)
-				localeChanged=True
-			except:
-				pass
-			if not localeChanged and '_' in lang:
-				#Python couldn'tsupport the language_country locale, just try language.
-				try:
-					locale.setlocale(locale.LC_ALL,lang.split('_')[0])
-				except:
-					pass
-			#Set the windows locale for this thread (NVDA core) to this locale.
-			LCID=localeNameToWindowsLCID(lang)
+	if lang == "Windows":
+		localeName = getWindowsLanguage()
+	else:
+		localeName = lang
+		# Set the windows locale for this thread (NVDA core) to this locale.
+		try:
+			LCID = localeNameToWindowsLCID(lang)
 			ctypes.windll.kernel32.SetThreadLocale(LCID)
+		except IOError:
+			log.debugWarning(f"couldn't set windows thread locale to {lang}")
+
+	try:
+		trans = gettext.translation("nvda", localedir="locale", languages=[localeName])
+		curLang = localeName
 	except IOError:
-		trans=gettext.translation("nvda",fallback=True)
-		curLang="en"
+		log.debugWarning(f"couldn't set the translation service locale to {localeName}")
+		trans = gettext.translation("nvda", fallback=True)
+		curLang = "en"
+
 	# #9207: Python 3.8 adds gettext.pgettext, so add it to the built-in namespace.
 	trans.install(names=["pgettext"])
+	setLocale(curLang)
+
+
+def setLocale(localeName: str) -> None:
+	'''
+	Set python's locale using a `localeName` such as "en", "ru_RU", or "es-ES".
+	Will fallback on `curLang` if it cannot be set and finally fallback to the system locale.
+	'''
+
+	r'''
+	Python 3.8's locale system allows you to set locales that you cannot get
+	so we must test for both ValueErrors and locale.Errors
+
+	>>> import locale
+	>>> locale.setlocale(locale.LC_ALL, 'foobar')
+	Traceback (most recent call last):
+	File "<stdin>", line 1, in <module>
+	File "Python38-32\lib\locale.py", line 608, in setlocale
+		return _setlocale(category, locale)
+	locale.Error: unsupported locale setting
+	>>> locale.setlocale(locale.LC_ALL, 'en-GB')
+	'en-GB'
+	>>> locale.getlocale()
+	Traceback (most recent call last):
+	File "<stdin>", line 1, in <module>
+	File "Python38-32\lib\locale.py", line 591, in getlocale
+		return _parse_localename(localename)
+	File "Python38-32\lib\locale.py", line 499, in _parse_localename
+		raise ValueError('unknown locale: %s' % localename)
+	ValueError: unknown locale: en-GB
+	'''
+	originalLocaleName = localeName
+	# Try setting Python's locale to localeName
+	try:
+		locale.setlocale(locale.LC_ALL, localeName)
+		locale.getlocale()
+		log.debug(f"set python locale to {localeName}")
+		return
+	except locale.Error:
+		log.debugWarning(f"python locale {localeName} could not be set")
+	except ValueError:
+		log.debugWarning(f"python locale {localeName} could not be retrieved with getlocale")
+
+	if '-' in localeName:
+		# Python couldn't support the language-country locale, try language_country.
+		try:
+			localeName = localeName.replace('-', '_')
+			locale.setlocale(locale.LC_ALL, localeName)
+			locale.getlocale()
+			log.debug(f"set python locale to {localeName}")
+			return
+		except locale.Error:
+			log.debugWarning(f"python locale {localeName} could not be set")
+		except ValueError:
+			log.debugWarning(f"python locale {localeName} could not be retrieved with getlocale")
+
+	if '_' in localeName:
+		# Python couldn't support the language_country locale, just try language.
+		try:
+			localeName = localeName.split('_')[0]
+			locale.setlocale(locale.LC_ALL, localeName)
+			locale.getlocale()
+			log.debug(f"set python locale to {localeName}")
+			return
+		except locale.Error:
+			log.debugWarning(f"python locale {localeName} could not be set")
+		except ValueError:
+			log.debugWarning(f"python locale {localeName} could not be retrieved with getlocale")
+
+	try:
+		locale.getlocale()
+	except ValueError:
+		# as the locale may have been changed to something that getlocale() couldn't retrieve
+		# reset to default locale
+		if originalLocaleName == curLang:
+			# reset to system locale default if we can't set the current lang's locale
+			locale.setlocale(locale.LC_ALL, "")
+			log.debugWarning(f"set python locale to system default")
+		else:
+			log.debugWarning(f"setting python locale to the current language {curLang}")
+			# fallback and try to reset the locale to the current lang
+			setLocale(curLang)
 
 
 def getLanguage() -> str:
@@ -314,5 +397,9 @@ def normalizeLanguage(lang):
 	134:'qut',
 	135:'rw',
 	136:'wo',
-	140:'gbz'
+	140: 'gbz',
+	1170: 'ckb',
+	1109: 'my',
+	1143: 'so',
+	9242: 'sr',
 }


Test patch:
diff --git a/tests/unit/test_languageHandler.py b/tests/unit/test_languageHandler.py
--- a/tests/unit/test_languageHandler.py
+++ b/tests/unit/test_languageHandler.py
@@ -1,20 +1,32 @@
-#tests/unit/test_languageHandler.py
-#A part of NonVisual Desktop Access (NVDA)
-#This file is covered by the GNU General Public License.
-#See the file COPYING for more details.
-#Copyright (C) 2017 NV Access Limited
+# A part of NonVisual Desktop Access (NVDA)
+# This file is covered by the GNU General Public License.
+# See the file COPYING for more details.
+# Copyright (C) 2017-2021 NV Access Limited
 
 """Unit tests for the languageHandler module.
 """
 
 import unittest
 import languageHandler
-from languageHandler import LCID_NONE
+from languageHandler import LCID_NONE, windowsPrimaryLCIDsToLocaleNames
+import locale
+import ctypes
 
 LCID_ENGLISH_US = 0x0409
+UNSUPPORTED_PYTHON_LOCALES = {
+	"an",
+	"ckb",
+	"kmr",
+	"mn",
+	"my",
+	"ne",
+	"so",
+}
+TRANSLATABLE_LANGS = set(l[0] for l in languageHandler.getAvailableLanguages()) - {"Windows"}
+WINDOWS_LANGS = set(locale.windows_locale.values()).union(windowsPrimaryLCIDsToLocaleNames.values())
 
-class TestLocaleNameToWindowsLCID(unittest.TestCase):
 
+class TestLocaleNameToWindowsLCID(unittest.TestCase):
 	def test_knownLocale(self):
 		lcid = languageHandler.localeNameToWindowsLCID("en")
 		self.assertEqual(lcid, LCID_ENGLISH_US)
@@ -31,3 +43,156 @@ def test_nonStandardLocale(self):
 	def test_invalidLocale(self):
 		lcid = languageHandler.localeNameToWindowsLCID("zzzz")
 		self.assertEqual(lcid, LCID_NONE)
+
+
+class Test_languageHandler_setLocale(unittest.TestCase):
+	"""Tests for the function languageHandler.setLocale"""
+
+	SUPPORTED_LOCALES = [("en", "en_US"), ("fa-IR", "fa_IR"), ("an-ES", "an_ES")]
+
+	def setUp(self):
+		"""
+		`setLocale` doesn't change `languageHandler.curLang`, so reset the locale using `setLanguage` to
+		the current language for each test.
+		"""
+		languageHandler.setLanguage(languageHandler.curLang)
+
+	@classmethod
+	def tearDownClass(cls):
+		"""
+		`setLocale` doesn't change `languageHandler.curLang`, so reset the locale using `setLanguage` to
+		the current language so the tests can continue normally.
+		"""
+		languageHandler.setLanguage(languageHandler.curLang)
+
+	def test_SupportedLocale_LocaleIsSet(self):
+		"""
+		Tests several locale formats that should result in an expected python locale being set.
+		"""
+		for localeName in self.SUPPORTED_LOCALES:
+			with self.subTest(localeName=localeName):
+				languageHandler.setLocale(localeName[0])
+				self.assertEqual(locale.getlocale()[0], localeName[1])
+
+	def test_PythonUnsupportedLocale_LocaleUnchanged(self):
+		"""
+		Tests several locale formats that python doesn't support which will result in a return to the
+		current locale
+		"""
+		original_locale = locale.getlocale()
+		for localeName in UNSUPPORTED_PYTHON_LOCALES:
+			with self.subTest(localeName=localeName):
+				languageHandler.setLocale(localeName)
+				self.assertEqual(locale.getlocale(), original_locale)
+
+	def test_NVDASupportedAndPythonSupportedLocale_LanguageCodeMatches(self):
+		"""
+	 	Tests all the translatable languages that NVDA shows in the user preferences
+		excludes the locales that python doesn't support, as the expected behaviour is different.
+		"""
+		for localeName in TRANSLATABLE_LANGS - UNSUPPORTED_PYTHON_LOCALES:
+			with self.subTest(localeName=localeName):
+				languageHandler.setLocale(localeName)
+				current_locale = locale.getlocale()
+
+				if localeName == "uk":
+					self.assertEqual(current_locale[0], "English_United Kingdom")
+				else:
+					pythonLang = current_locale[0].split("_")[0]
+					langOnly = localeName.split("_")[0]
+					self.assertEqual(
+						langOnly,
+						pythonLang,
+						f"full values: {localeName} {current_locale[0]}",
+					)
+
+	def test_WindowsLang_LocaleCanBeRetrieved(self):
+		"""
+		We don't know whether python supports a specific windows locale so just ensure locale isn't
+		broken after testing these values.
+		"""
+		for localeName in WINDOWS_LANGS:
+			with self.subTest(localeName=localeName):
+				languageHandler.setLocale(localeName)
+				locale.getlocale()
+
+
+class Test_LanguageHandler_SetLanguage(unittest.TestCase):
+	"""Tests for the function languageHandler.setLanguage"""
+
+	UNSUPPORTED_WIN_LANGUAGES = ["an", "kmr"]
+
+	def tearDown(self):
+		"""
+		Resets the language to whatever it was before the testing suite begun.
+		"""
+		languageHandler.setLanguage(self._prevLang)
+
+	def __init__(self, *args, **kwargs):
+		self._prevLang = languageHandler.getLanguage()
+
+		ctypes.windll.kernel32.SetThreadLocale(0)
+		defaultThreadLocale = ctypes.windll.kernel32.GetThreadLocale()
+		self._defaultThreadLocaleName = languageHandler.windowsLCIDToLocaleName(
+			defaultThreadLocale
+		)
+
+		locale.setlocale(locale.LC_ALL, "")
+		self._defaultPythonLocale = locale.getlocale()
+
+		languageHandler.setLanguage(self._prevLang)
+		super().__init__(*args, **kwargs)
+
+	def test_NVDASupportedLanguages_LanguageIsSetCorrectly(self):
+		"""
+		Tests languageHandler.setLanguage, using all NVDA supported languages, which should do the following:
+		- set the translation service and languageHandler.curLang
+		- set the windows locale for the thread (fallback to system default)
+		- set the python locale for the thread (match the translation service, fallback to system default)
+		"""
+		for localeName in TRANSLATABLE_LANGS:
+			with self.subTest(localeName=localeName):
+				langOnly = localeName.split("_")[0]
+				languageHandler.setLanguage(localeName)
+				# check curLang/translation service is set
+				self.assertEqual(languageHandler.curLang, localeName)
+
+				# check Windows thread is set
+				threadLocale = ctypes.windll.kernel32.GetThreadLocale()
+				threadLocaleName = languageHandler.windowsLCIDToLocaleName(threadLocale)
+				threadLocaleLang = threadLocaleName.split("_")[0]
+				if localeName in self.UNSUPPORTED_WIN_LANGUAGES:
+					# our translatable locale isn't supported by windows
+					# check that the system locale is unchanged
+					self.assertEqual(self._defaultThreadLocaleName, threadLocaleName)
+				else:
+					# check that the language codes are correctly set for the thread
+					self.assertEqual(
+						langOnly,
+						threadLocaleLang,
+						f"full values: {localeName} {threadLocaleName}",
+					)
+
+				# check that the python locale is set
+				python_locale = locale.getlocale()
+				if localeName in UNSUPPORTED_PYTHON_LOCALES:
+					# our translatable locale isn't supported by python
+					# check that the system locale is unchanged
+					self.assertEqual(self._defaultPythonLocale, python_locale)
+				elif localeName == "uk":
+					self.assertEqual(python_locale[0], "English_United Kingdom")
+				else:
+					# check that the language codes are correctly set for python
+					pythonLang = python_locale[0].split("_")[0]
+					self.assertEqual(
+						langOnly, pythonLang, f"full values: {localeName} {python_locale}"
+					)
+
+	def test_WindowsLanguages_NoErrorsThrown(self):
+		"""
+		We don't know whether python or our translator system supports a specific windows locale
+		so just ensure the setLanguage process doesn't fail.
+		"""
+		for localeName in WINDOWS_LANGS:
+			with self.subTest(localeName=localeName):
+				languageHandler.setLanguage(localeName)