Repository: nautobot/nautobot

Issue:
Need to clean up properly when a task exceeds the Celery time limit
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version:
* Nautobot version:

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Configure a short `CELERY_TASK_TIME_LIMIT` value in `nautobot_config.py` (or implement a Job that will run for over 30 minutes before completion, since our current default `CELERY_TASK_TIME_LIMIT` is 1800 seconds or 30 minutes)
2. Run the Job
3. Follow the `nautobot-server celery worker` logs and observe that the job is terminated with a SIGKILL when it exceeds the configured task time limit.
4. Observe that the JobResult remains in `Running` state indefinitely and does not show that the job is terminated or errored.

<!-- What did you expect to happen? -->
### Expected Behavior

JobResult should switch to `ERRORED` state when the task is killed and include appropriate error information in its log.

<!-- What happened instead? -->
### Observed Behavior

JobResult remains in `Running` state indefinitely and does not show that the job is terminated or errored.

Probably we should add a `CELERY_TASK_SOFT_TIME_LIMIT` setting, as exceeding the soft time limit results in a `SoftTimeLImitExceeded` exception being thrown instead of a SIGKILL, which is much harder to catch and process gracefully.


Discussion:

Resolving patch:
diff --git a/nautobot/core/settings.py b/nautobot/core/settings.py
--- a/nautobot/core/settings.py
+++ b/nautobot/core/settings.py
@@ -526,8 +526,11 @@
 # Instruct celery to report the started status of a job, instead of just `pending`, `finished`, or `failed`
 CELERY_TASK_TRACK_STARTED = True
 
-# Global task time limit (seconds)
-CELERY_TASK_TIME_LIMIT = int(os.getenv("NAUTOBOT_CELERY_TASK_TIME_LIMIT", 30 * 60))
+# Global task time limits (seconds)
+# Exceeding the soft limit will result in a SoftTimeLimitExceeded exception,
+# while exceeding the hard limit will result in a SIGKILL.
+CELERY_TASK_SOFT_TIME_LIMIT = int(os.getenv("NAUTOBOT_CELERY_TASK_SOFT_TIME_LIMIT", 5 * 60))
+CELERY_TASK_TIME_LIMIT = int(os.getenv("NAUTOBOT_CELERY_TASK_TIME_LIMIT", 10 * 60))
 
 # The Redis connection defined in the CACHES config above for the broker and results backend
 CELERY_BROKER_URL = CACHES["default"]["LOCATION"]
diff --git a/nautobot/extras/models/customfields.py b/nautobot/extras/models/customfields.py
--- a/nautobot/extras/models/customfields.py
+++ b/nautobot/extras/models/customfields.py
@@ -4,6 +4,7 @@
 from datetime import datetime, date
 
 from django import forms
+from django.db import transaction
 from django.contrib.contenttypes.models import ContentType
 from django.core.serializers.json import DjangoJSONEncoder
 from django.core.validators import RegexValidator, ValidationError
@@ -494,7 +495,9 @@ def save(self, *args, **kwargs):
         super().save(*args, **kwargs)
 
         if self.value != database_object.value:
-            update_custom_field_choice_data.delay(self.field.pk, database_object.value, self.value)
+            transaction.on_commit(
+                lambda: update_custom_field_choice_data.delay(self.field.pk, database_object.value, self.value)
+            )
 
     def delete(self, *args, **kwargs):
         """
diff --git a/nautobot/extras/signals.py b/nautobot/extras/signals.py
--- a/nautobot/extras/signals.py
+++ b/nautobot/extras/signals.py
@@ -8,6 +8,7 @@
 from cacheops.signals import cache_invalidated, cache_read
 from django.conf import settings
 from django.contrib.contenttypes.models import ContentType
+from django.db import transaction
 from django.db.models.signals import m2m_changed, pre_delete
 from django.dispatch import receiver
 from django.utils import timezone
@@ -106,11 +107,11 @@ def handle_cf_removed_obj_types(instance, action, pk_set, **kwargs):
     """
     if action == "post_remove":
         # Existing content types have been removed from the custom field, delete their data
-        delete_custom_field_data.delay(instance.name, pk_set)
+        transaction.on_commit(lambda: delete_custom_field_data.delay(instance.name, pk_set))
 
     elif action == "post_add":
         # New content types have been added to the custom field, provision them
-        provision_field.delay(instance.pk, pk_set)
+        transaction.on_commit(lambda: provision_field.delay(instance.pk, pk_set))
 
 
 m2m_changed.connect(handle_cf_removed_obj_types, sender=CustomField.content_types.through)


Test patch: