Repository: netbox-community/netbox

Issue:
Race condition caused by PR #7803 when clearing modules from the sys.modules cache
### NetBox version

v3.1.9

### Python version

3.9

### Steps to Reproduce

As race conditions a finicky in nature, I can't really guarantee the reproduction steps.

1. Create any script and put in the scripts folder.
2. Create a user
3. Create an API token for the user
4. Run the following bash script:
```
for y in `seq 1 4` ; \
    do (for x in `seq 1 4` ; \
        do curl -H "Authorization: Token YOUR_TOKEN" -H "Accept: application/json" http://127.0.0.1:8000/api/extras/scripts/ -I ; \
    done)& ; \
done
```

### Expected Behavior

No exceptions

### Observed Behavior

```
Internal Server Error: /api/extras/scripts/
Traceback (most recent call last):
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/user/devel/netbox-source/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/user/devel/netbox-source/netbox/extras/api/views.py", line 295, in list
    for script_list in get_scripts().values():
  File "/home/user/devel/netbox-source/netbox/extras/scripts.py", line 495, in get_scripts
    module = importer.find_module(module_name).load_module(module_name)
  File "<frozen importlib._bootstrap_external>", line 548, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1063, in load_module
  File "<frozen importlib._bootstrap_external>", line 888, in load_module
  File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 719, in _load
  File "<frozen importlib._bootstrap>", line 699, in _load_unlocked
KeyError: 'ScriptName'
```

The reason is that get_scripts first clears the cache and then loads the module:

https://github.com/netbox-community/netbox/blob/e09ab79a1ac508779b715f5c7fb00412d1b57308/netbox/extras/scripts.py#L501-L504

If another thread is in the process of loading the modules while another thread is clearing the cache importlib will fail. (I assume this is the issue).

Not sure how to fix it, other than putting a lock around the cache clearing and loading of the module.


Discussion:
The issue is obviously only hit when running a threaded application server. In normal use the issue is very rarely hit and when it is, it mostly just results in a 500 returned for the periodic check for script completion. No error is shown to the user and the script succeeds like normal.

Should be fixed regardless. I hope we can find a solution that allows the functionality to remain, as it was quite annoying to have to restart netbox if you renamed a script class.

Resolving patch:
diff --git a/netbox/extras/scripts.py b/netbox/extras/scripts.py
--- a/netbox/extras/scripts.py
+++ b/netbox/extras/scripts.py
@@ -5,6 +5,7 @@
 import pkgutil
 import sys
 import traceback
+import threading
 from collections import OrderedDict
 
 import yaml
@@ -42,6 +43,8 @@
     'TextVar',
 ]
 
+lock = threading.Lock()
+
 
 #
 # Script variables
@@ -491,11 +494,14 @@ def get_scripts(use_names=False):
     # Iterate through all modules within the scripts path. These are the user-created files in which reports are
     # defined.
     for importer, module_name, _ in pkgutil.iter_modules([settings.SCRIPTS_ROOT]):
-        # Remove cached module to ensure consistency with filesystem
-        if module_name in sys.modules:
-            del sys.modules[module_name]
+        # Use a lock as removing and loading modules is not thread safe
+        with lock:
+            # Remove cached module to ensure consistency with filesystem
+            if module_name in sys.modules:
+                del sys.modules[module_name]
+
+            module = importer.find_module(module_name).load_module(module_name)
 
-        module = importer.find_module(module_name).load_module(module_name)
         if use_names and hasattr(module, 'name'):
             module_name = module.name
         module_scripts = OrderedDict()


Test patch: