Repository: nautobot/nautobot

Issue:
Deadlock on get next prefix API
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable):
* Python version: 3.8
* Database platform, version: 1.3.8 & 1.4.1
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Perform API call for get next prefix
2. abruptly stop Nautobot service before call can complete
3. Restart server

<!-- What did you expect to happen? -->
### Expected Behavior
Lock ages out

<!-- What happened instead? -->
### Observed Behavior
Places Nautobot in a deadlock state for get-next-prefix


### Proposed Fix
https://github.com/nautobot/nautobot/blob/v1.4.1/nautobot/ipam/api/views.py#L143
Set a configurable timeout that defaults to 300 seconds. A user should never be able to hit 5 minutes in an API call which would safely add the safe measure. And after 5 minutes if the lock is not gracefully released it will be aged out.


Discussion:

Resolving patch:
diff --git a/nautobot/core/settings.py b/nautobot/core/settings.py
--- a/nautobot/core/settings.py
+++ b/nautobot/core/settings.py
@@ -76,7 +76,6 @@
 HTTP_PROXIES = None
 JOBS_ROOT = os.getenv("NAUTOBOT_JOBS_ROOT", os.path.join(NAUTOBOT_ROOT, "jobs").rstrip("/"))
 MAINTENANCE_MODE = False
-
 # Metrics
 METRICS_ENABLED = False
 
@@ -645,6 +644,10 @@
 
 CELERY_BEAT_SCHEDULER = "nautobot.core.celery.schedulers:NautobotDatabaseScheduler"
 
+# Sets an age out timer of redis lock. This is NOT implicitially applied to locks, must be added
+# to a lock creation as `timeout=settings.REDIS_LOCK_TIMEOUT`
+REDIS_LOCK_TIMEOUT = int(os.getenv("NAUTOBOT_REDIS_LOCK_TIMEOUT", "600"))
+
 #
 # Custom branding (logo and title)
 #
diff --git a/nautobot/ipam/api/views.py b/nautobot/ipam/api/views.py
--- a/nautobot/ipam/api/views.py
+++ b/nautobot/ipam/api/views.py
@@ -1,3 +1,4 @@
+from django.conf import settings
 from django.core.cache import cache
 from django.shortcuts import get_object_or_404
 from drf_spectacular.utils import extend_schema, extend_schema_view
@@ -140,7 +141,7 @@ def available_prefixes(self, request, pk=None):
         prefix = get_object_or_404(self.queryset, pk=pk)
         if request.method == "POST":
 
-            with cache.lock("available-prefixes", blocking_timeout=5):
+            with cache.lock("available-prefixes", blocking_timeout=5, timeout=settings.REDIS_LOCK_TIMEOUT):
                 available_prefixes = prefix.get_available_prefixes()
 
                 # Validate Requested Prefixes' length
@@ -226,7 +227,7 @@ def available_ips(self, request, pk=None):
         # Create the next available IP within the prefix
         if request.method == "POST":
 
-            with cache.lock("available-ips", blocking_timeout=5):
+            with cache.lock("available-ips", blocking_timeout=5, timeout=settings.REDIS_LOCK_TIMEOUT):
 
                 # Normalize to a list of objects
                 requested_ips = request.data if isinstance(request.data, list) else [request.data]


Test patch: