Repository: nvaccess/nvda

Issue:
An opened file prevents add-on removal in the future
### Steps to reproduce:

Example with the Character Information add-on
1. Open a file of the add-on in an editor, e.g. `%appdata%\nvda\addons\charInfo\globalPlugins\charinfo\__init__.py`
2. Open Add-on manager and remove Character Information add-on
   The status of the add-on is "Enabled, Removed after restart"
3. Close add-on manager and restart NVDA as asked.
4. Open add-on manager
   The status of the add-on is "Enabled, Removed after restart". This is expected because the add-on could not be removed due to the file opened in the editor.
5. Close the file of the add-on opened in the editor.
6. Close the add-on manager and restart NVDA as asked.
7. Open add-on manager and check the add-on's status
8. Execute steps 5. to 7. as many times as you want.

### Actual behavior:
At step 7., the status of the add-on is "Enabled, Removed after restart".
The only way I have found to remove this add-on is to remove manually its folder or to reinstall it before removing it again.

### Expected behavior:
At step 7., the add-on should not appear anymore in the list since nothing prevents NVDA anymore to remove the add-on after step 6.


### System configuration
#### NVDA installed/portable/running from source:
Installed

#### NVDA version:
2021.1rc2
#### Windows version:
Windows 10 1809 (64-bit) build 17763.1935

#### Name and version of other software in use when reproducing the issue:
N/A
#### Other information about your system:
N/A
### Other questions
#### Does the issue still occur after restarting your computer?
Yes
#### Have you tried any other versions of NVDA? If so, please report their behaviors.
Yes, this error has always been present, surely in all versions of NVDA 2019.3 and beyond.


#### If add-ons are disabled, is your problem still occurring?
N/A
#### Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Yes



Discussion:
Yes, I have a feeling that  a second file is involved in setting the state 
and it is now going to contradict itself until those settings agree. It 
fails safe in that it does not disable anything, as you say, it has to be 
reinstalled then removed the legal way again. Its been like that for some 
time. Not sure if its deliberate or not, but I'm sure somebody will 
remember.
 Brian

***@***.***
Sent via blueyonder.
Please address personal E-mail to:-
***@***.***, putting 'Brian Gaff'
in the display name field.


I cannot reproduce this with 2021.1 nor with latest Alpha running from sources. At first I thought this might be specific to Char Info but its not reproducible with it either. @CyrilleB79 Perhaps you've forgot to close the folder from which you've started the editor - this happened to me several times in the past and  caused add-on removal process to fail.
@lukaszgo1 I cannot re-test for now.
Anyway if Windows Explorer is opened at step 1. along with the file but closed at step 5., the add-on may not be removed at step 2. but should be removed at step 6. in any case. Do you agree?
Would it be reproducible on your side keeping opened not only the file but also Windows Explorer before step 5.?

I agree that add-on should be removed as soon as both the file in the editor and Windows Explorer opened in the add-on folder are closed and that is already the case for me in the latest Alpha / 2021.1.
@CyrilleB79 could it be possible that the process of the editor is still running in the task manager after closing the file? Did you check that?
Retesting it, I cannot reproduce with the exact same steps but I can reproduce with slightly differents steps. To simplify the issue and avoid phantom processes, I do not open any file in the editor but only keep a subfolder of the add-on opened in the Explorer, e.g. the doc subfolder.
And instead of only removing the add-on, I re-install (or update) an add-on that is already installed.
This may happen with any add-on and not only charInfo. This time, I have tested with sayProductNameAndVersion.

@lukaszgo1 could you try this? Thanks.



Yeah - I can reproduce with your latest  STR  - caused by a very simple coding mistake. I'll submit a PR for this and some other issues related to the add-on removal which I've found along the way when testing later today or tomorrow.

Resolving patch:
diff --git a/source/addonHandler/__init__.py b/source/addonHandler/__init__.py
--- a/source/addonHandler/__init__.py
+++ b/source/addonHandler/__init__.py
@@ -1,8 +1,6 @@
-# -*- coding: UTF-8 -*-
-# addonHandler.py
 # A part of NonVisual Desktop Access (NVDA)
-# Copyright (C) 2012-2019 Rui Batista, NV Access Limited, Noelia Ruiz Martínez,
-# Joseph Lee, Babbage B.V., Arnold Loubriat
+# Copyright (C) 2012-2021 Rui Batista, NV Access Limited, Noelia Ruiz Martínez,
+# Joseph Lee, Babbage B.V., Arnold Loubriat, Łukasz Golonka
 # This file is covered by the GNU General Public License.
 # See the file COPYING for more details.
 
@@ -18,19 +16,21 @@
 from io import StringIO
 import pickle
 from six import string_types
+import typing
 import globalVars
 import zipfile
 from configobj import ConfigObj
 from configobj.validate import Validator
 
 import config
-import globalVars
 import languageHandler
 from logHandler import log
 import winKernel
 import addonAPIVersion
 from . import addonVersionCheck
 from .addonVersionCheck import isAddonCompatible
+import buildVersion
+
 
 MANIFEST_FILENAME = "manifest.ini"
 stateFilename="addonsState.pickle"
@@ -40,42 +40,87 @@
 ADDON_PENDINGINSTALL_SUFFIX=".pendingInstall"
 DELETEDIR_SUFFIX=".delete"
 
-state={}
-
 # Add-ons that are blocked from running because they are incompatible
 _blockedAddons=set()
 
-def loadState():
-	global state
-	statePath=os.path.join(globalVars.appArgs.configPath,stateFilename)
-	try:
-		# #9038: Python 3 requires binary format when working with pickles.
-		with open(statePath, "rb") as f:
-			state = pickle.load(f)
-		if "disabledAddons" not in state:
-			state["disabledAddons"] = set()
-		if "pendingDisableSet" not in state:
-			state["pendingDisableSet"] = set()
-		if "pendingEnableSet" not in state:
-			state["pendingEnableSet"] = set()
-	except:
-		# Defaults.
-		state = {
-			"pendingRemovesSet":set(),
-			"pendingInstallsSet":set(),
-			"disabledAddons":set(),
-			"pendingEnableSet":set(),
-			"pendingDisableSet":set(),
-		}
-
-def saveState():
-	statePath=os.path.join(globalVars.appArgs.configPath,stateFilename)
-	try:
-		# #9038: Python 3 requires binary format when working with pickles.
-		with open(statePath, "wb") as f:
-			pickle.dump(state, f, protocol=0)
-	except:
-		log.debugWarning("Error saving state", exc_info=True)
+
+class AddonsState(collections.UserDict):
+	"""Subclasses `collections.UserDict` to preserve backwards compatibility."""
+
+	_DEFAULT_STATE_CONTENT = {
+		"pendingRemovesSet": set(),
+		"pendingInstallsSet": set(),
+		"disabledAddons": set(),
+		"pendingEnableSet": set(),
+		"pendingDisableSet": set(),
+	}
+
+	@property
+	def statePath(self) -> os.PathLike:
+		"""Returns path to the state file. """
+		return os.path.join(globalVars.appArgs.configPath, stateFilename)
+
+	def load(self) -> None:
+		"""Populates state with the default content and then loads values from the config."""
+		self.update(self._DEFAULT_STATE_CONTENT)
+		try:
+			# #9038: Python 3 requires binary format when working with pickles.
+			with open(self.statePath, "rb") as f:
+				state = pickle.load(f)
+				self.update(state)
+		except FileNotFoundError:
+			pass  # Clean config - no point logging in this case
+		except IOError:
+			log.debug("Error when reading state file", exc_info=True)
+		except pickle.UnpicklingError:
+			log.debugWarning("Failed to unpickle state", exc_info=True)
+
+	def removeStateFile(self) -> None:
+		try:
+			os.remove(self.statePath)
+		except FileNotFoundError:
+			pass  # Probably clean config - no point in logging in this case.
+		except OSError:
+			log.error(f"Failed to remove state file {self.statePath}", exc_info=True)
+
+	def save(self) -> None:
+		"""Saves content of the state to a file unless state is empty in which case this would be pointless."""
+		if any(self.values()):
+			try:
+				# #9038: Python 3 requires binary format when working with pickles.
+				with open(self.statePath, "wb") as f:
+					# We cannot pickle instance of `AddonsState` directly
+					# since older versions of NVDA aren't aware about this clas and they're expecting state
+					# to be a standard `dict`.
+					pickle.dump(self.data, f, protocol=0)
+			except (IOError, pickle.PicklingError):
+				log.debugWarning("Error saving state", exc_info=True)
+		else:
+			# Empty state - just delete state file and don't save anything.
+			self.removeStateFile()
+
+	def cleanupRemovedDisabledAddons(self) -> None:
+		"""Versions of NVDA before #12792 failed to remove add-on from list of disabled add-ons
+		during uninstallation. As a result after reinstalling add-on with the same name it was disabled
+		by default confusing users. Fix this by removing all add-ons no longer present in the config
+		from the list of disabled add-ons in the state."""
+		installedAddonNames = tuple(a.name for a in getAvailableAddons())
+		for disabledAddonName in list(self["disabledAddons"]):
+			if disabledAddonName not in installedAddonNames:
+				self["disabledAddons"].discard(disabledAddonName)
+
+
+state = AddonsState()
+
+
+# Deprecated - use `state.save` and `state.load` instead.
+if buildVersion.version_year < 2022:
+	def saveState():
+		state.save()
+
+	def loadState():
+		state.load()
+
 
 def getRunningAddons():
 	""" Returns currently loaded add-ons.
@@ -96,53 +141,21 @@ def getIncompatibleAddons(
 		)
 	))
 
-def completePendingAddonRemoves():
-	"""Removes any add-ons that could not be removed on the last run of NVDA"""
-	user_addons = os.path.join(globalVars.appArgs.configPath, "addons")
-	pendingRemovesSet=state['pendingRemovesSet']
-	for addonName in list(pendingRemovesSet):
-		addonPath=os.path.join(user_addons,addonName)
-		if os.path.isdir(addonPath):
-			addon=Addon(addonPath)
-			try:
-				addon.completeRemove()
-			except RuntimeError:
-				log.exception("Failed to remove %s add-on"%addonName)
-				continue
-		pendingRemovesSet.discard(addonName)
 
-def completePendingAddonInstalls():
-	user_addons = os.path.join(globalVars.appArgs.configPath, "addons")
-	pendingInstallsSet=state['pendingInstallsSet']
-	for addonName in pendingInstallsSet:
-		newPath=os.path.join(user_addons,addonName)
-		oldPath=newPath+ADDON_PENDINGINSTALL_SUFFIX
-		try:
-			os.rename(oldPath,newPath)
-		except:
-			log.error("Failed to complete addon installation for %s"%addonName,exc_info=True)
-	pendingInstallsSet.clear()
+def removeFailedDeletion(path: os.PathLike):
+	shutil.rmtree(path, ignore_errors=True)
+	if os.path.exists(path):
+		log.error(f"Failed to delete path {path}, try removing manually")
 
-def removeFailedDeletions():
-	user_addons = os.path.join(globalVars.appArgs.configPath, "addons")
-	for p in os.listdir(user_addons):
-		if p.endswith(DELETEDIR_SUFFIX):
-			path=os.path.join(user_addons,p)
-			shutil.rmtree(path,ignore_errors=True)
-			if os.path.exists(path):
-				log.error("Failed to delete path %s, try removing manually"%path)
 
-_disabledAddons = set()
 def disableAddonsIfAny():
 	"""
 	Disables add-ons if told to do so by the user from add-ons manager.
 	This is usually executed before refreshing the list of available add-ons.
 	"""
-	global _disabledAddons
 	# Pull in and enable add-ons that should be disabled and enabled, respectively.
 	state["disabledAddons"] |= state["pendingDisableSet"]
 	state["disabledAddons"] -= state["pendingEnableSet"]
-	_disabledAddons = state["disabledAddons"]
 	state["pendingDisableSet"].clear()
 	state["pendingEnableSet"].clear()
 
@@ -151,14 +164,12 @@ def initialize():
 	if config.isAppX:
 		log.info("Add-ons not supported when running as a Windows Store application")
 		return
-	loadState()
-	removeFailedDeletions()
-	completePendingAddonRemoves()
-	completePendingAddonInstalls()
+	state.load()
 	# #3090: Are there add-ons that are supposed to not run for this session?
 	disableAddonsIfAny()
-	getAvailableAddons(refresh=True)
-	saveState()
+	getAvailableAddons(refresh=True, isFirstLoad=True)
+	state.cleanupRemovedDisabledAddons()
+	state.save()
 
 
 def terminate():
@@ -176,7 +187,8 @@ def _getDefaultAddonPaths():
 		addon_paths.append(user_addons)
 	return addon_paths
 
-def _getAvailableAddonsFromPath(path):
+
+def _getAvailableAddonsFromPath(path, isFirstLoad=False):
 	""" Gets available add-ons from path.
 	An addon is only considered available if the manifest file is loaded with no errors.
 	@param path: path from where to find addon directories.
@@ -185,7 +197,10 @@ def _getAvailableAddonsFromPath(path):
 	"""
 	log.debug("Listing add-ons from %s", path)
 	for p in os.listdir(path):
-		if p.endswith(DELETEDIR_SUFFIX): continue
+		if p.endswith(DELETEDIR_SUFFIX):
+			if isFirstLoad:
+				removeFailedDeletion(os.path.join(path, p))
+			continue
 		addon_path = os.path.join(path, p)
 		if os.path.isdir(addon_path) and addon_path not in ('.', '..'):
 			if not len(os.listdir(addon_path)):
@@ -195,6 +210,23 @@ def _getAvailableAddonsFromPath(path):
 				try:
 					a = Addon(addon_path)
 					name = a.manifest['name']
+					if (
+						isFirstLoad
+						and name in state["pendingRemovesSet"]
+						and not a.path.endswith(ADDON_PENDINGINSTALL_SUFFIX)
+					):
+						try:
+							a.completeRemove()
+						except RuntimeError:
+							log.exception(f"Failed to remove {name} add-on")
+						continue
+					if(
+						isFirstLoad
+						and (name in state["pendingInstallsSet"] or a.path.endswith(ADDON_PENDINGINSTALL_SUFFIX))
+					):
+						newPath = a.completeInstall()
+						if newPath:
+							a = Addon(newPath)
 					log.debug(
 						"Found add-on {name} - {a.version}."
 						" Requires API: {a.minimumNVDAVersion}."
@@ -210,23 +242,28 @@ def _getAvailableAddonsFromPath(path):
 					yield a
 				except:
 					log.error("Error loading Addon from path: %s", addon_path, exc_info=True)
-				
+
 _availableAddons = collections.OrderedDict()
-def getAvailableAddons(refresh=False, filterFunc=None):
+
+
+def getAvailableAddons(
+		refresh: bool = False,
+		filterFunc: typing.Optional[typing.Callable[["Addon"], bool]] = None,
+		isFirstLoad: bool = False
+) -> typing.Generator["Addon", None, None]:
 	""" Gets all available addons on the system.
 	@param refresh: Whether or not to query the file system for available add-ons.
-	@type refresh: bool
 	@param filterFunc: A function that allows filtering of add-ons.
-		It takes an L{Addon} as its only argument
-		and returns a C{bool} indicating whether the add-on matches the provided filter.
-	@type filterFunc: callable
-	@rtype generator of Addon instances.
+	It takes an L{Addon} as its only argument
+	and returns a C{bool} indicating whether the add-on matches the provided filter.
+	: isFirstLoad: Should add-ons that are pending installations / removal from the file system
+	be installed / removed.
 	"""
 	if filterFunc and not callable(filterFunc):
 		raise TypeError("The provided filterFunc is not callable")
 	if refresh:
 		_availableAddons.clear()
-		generators = [_getAvailableAddonsFromPath(path) for path in _getDefaultAddonPaths()]
+		generators = [_getAvailableAddonsFromPath(path, isFirstLoad) for path in _getDefaultAddonPaths()]
 		for addon in itertools.chain(*generators):
 			_availableAddons[addon.path] = addon
 	return (addon for addon in _availableAddons.values() if not filterFunc or filterFunc(addon))
@@ -247,7 +284,7 @@ def installAddonBundle(bundle):
 		addon.completeRemove(runUninstallTask=False)
 		raise AddonError("Installation failed")
 	state['pendingInstallsSet'].add(bundle.manifest['name'])
-	saveState()
+	state.save()
 	return addon
 
 class AddonError(Exception):
@@ -307,6 +344,16 @@ def isPendingRemove(self):
 		"""True if this addon is marked for removal."""
 		return not self.isPendingInstall and self.name in state['pendingRemovesSet']
 
+	def completeInstall(self):
+		newPath = self.path.replace(ADDON_PENDINGINSTALL_SUFFIX, "")
+		oldPath = self.path
+		try:
+			os.rename(oldPath, newPath)
+			state['pendingInstallsSet'].discard(self.name)
+			return newPath
+		except OSError:
+			log.error(f"Failed to complete addon installation for {self.name}", exc_info=True)
+
 	def requestRemove(self):
 		"""Markes this addon for removal on NVDA restart."""
 		if self.isPendingInstall:
@@ -317,10 +364,10 @@ def requestRemove(self):
 		else:
 			state['pendingRemovesSet'].add(self.name)
 			# There's no point keeping a record of this add-on pending being disabled now.
-			# However, if the addon is in _disabledAddons, then it needs to stay there so that
+			# However, if the addon is disabled, then it needs to remain disabled so that
 			# the status in addonsManager continues to say "disabled"
 			state['pendingDisableSet'].discard(self.name)
-		saveState()
+		state.save()
 
 	def completeRemove(self,runUninstallTask=True):
 		if runUninstallTask:
@@ -343,10 +390,11 @@ def completeRemove(self,runUninstallTask=True):
 			log.error("Error removing addon directory %s, deferring until next NVDA restart"%self.path)
 		# clean up the addons state. If an addon with the same name is installed, it should not be automatically
 		# disabled / blocked.
-		log.debug("removing addon {} from _disabledAddons/_blockedAddons".format(self.name))
-		_disabledAddons.discard(self.name)
+		log.debug(f"removing addon {self.name} from the list of disabled / blocked add-ons")
+		state["disabledAddons"].discard(self.name)
+		state['pendingRemovesSet'].discard(self.name)
 		_blockedAddons.discard(self.name)
-		saveState()
+		state.save()
 
 	def addToPackagePath(self, package):
 		""" Adds this L{Addon} extensions to the specific package path if those exist.
@@ -361,7 +409,7 @@ def addToPackagePath(self, package):
 		"""
 		# #3090: Ensure that we don't add disabled / blocked add-ons to package path.
 		# By returning here the addon does not "run"/ become active / registered.
-		if self.isDisabled or self.isBlocked:
+		if self.isDisabled or self.isBlocked or self.isPendingInstall:
 			return
 
 		extension_path = os.path.join(self.path, package.__name__)
@@ -402,7 +450,7 @@ def enable(self, shouldEnable):
 			elif self.name not in state["disabledAddons"]:
 				state["pendingDisableSet"].add(self.name)
 		# Record enable/disable flags as a way of preparing for disaster such as sudden NVDA crash.
-		saveState()
+		state.save()
 
 	@property
 	def isRunning(self):
@@ -410,7 +458,7 @@ def isRunning(self):
 
 	@property
 	def isDisabled(self):
-		return self.name in _disabledAddons
+		return self.name in state["disabledAddons"]
 
 	@property
 	def isBlocked(self):
@@ -508,7 +556,6 @@ def getCodeAddon(obj=None, frameDist=1):
 	@return: L{Addon} instance or None if no code does not belong to a add-on package.
 	@rtype: C{Addon}
 	"""
-	global _availableAddons
 	if obj is None:
 		obj = sys._getframe(frameDist)
 	fileName  = inspect.getfile(obj)


Test patch: