Repository: nautobot/nautobot

Issue:
kombu.exceptions.EncodeError when trying to execute Job
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.8.10
* Nautobot version: 1.1.0b1

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1.  Copy [Planned Site Job](https://nautobot.readthedocs.io/en/latest/additional-features/jobs/#creating-objects-for-a-planned-site) from Nautobot docs to  a `.py` file `$NAUTOBOT_ROOT/jobs/` 
2. In Web UI, Extensibility-->Jobs
3. Run the `New Branch` job

<!-- What did you expect to happen? -->
### Expected Behavior
Job to execute


<!-- What happened instead? -->
### Observed Behavior
Got the following message in the UI:

 **Server Error**
**There was a problem with your request. Please contact an administrator.**

**The complete exception is provided below:**

```
<class 'kombu.exceptions.EncodeError'>

Object of type DeviceType is not JSON serializable

Python version: 3.8.10
Nautobot version: 1.1.0b1
```



Discussion:
Oof this one is ugly. I am able to reliably reproduce it and attempts to get nested serialization of model objects using JSON and the `NautobotKomboJsonDecoder` class will alleviate issues with *our* objects such as `DeviceType`, but ultimately results the same error but with `ModelState`:

```
Object of type ModelState is not JSON serializable
```

This is `django.db.models.base.ModelState` which is the internally cached state of the model. I've never seen this leak before. 

I can completely obviate this error scenario entirely by hard-coding these settings in `nautobot_config.py`:

```python
CELERY_ACCEPT_CONTENT = {"pickle"}
CELERY_RESULT_ACCEPT_CONTENT = CELERY_ACCEPT_CONTENT
CELERY_TASK_SERIALIZER = "pickle"
CELERY_RESULT_SERIALIZER = "pickle"
```

This forces Celery to use Pickle. This isn't necessarily desirable, but I think we have two options:

1. Go with Pickle for now while we sort it out
2. Try to get to the bottom of how to do nested JSON serialize/deserialize for messaging in Celery
Given the current feature set in 1.1, pickle would work fine. However the switch to json is driven by the scheduler because args and kwargs have to be stored in the DB. So in a pinch, to get 1.1 we could switch it back to pickle and then figure out what the issue is.

Resolving patch:
diff --git a/nautobot/extras/api/views.py b/nautobot/extras/api/views.py
--- a/nautobot/extras/api/views.py
+++ b/nautobot/extras/api/views.py
@@ -277,10 +277,6 @@ def run(self, request, class_path):
         if not request.user.has_perm("extras.run_job"):
             raise PermissionDenied("This user does not have permission to run jobs.")
 
-        # Check that at least one RQ worker is running
-        if not Worker.count(get_connection("default")):
-            raise RQWorkerNotRunningException()
-
         job_class = self._get_job_class(class_path)
         job = job_class()
 
diff --git a/nautobot/extras/jobs.py b/nautobot/extras/jobs.py
--- a/nautobot/extras/jobs.py
+++ b/nautobot/extras/jobs.py
@@ -15,6 +15,9 @@
 from django.conf import settings
 from django.core.validators import RegexValidator
 from django.db import transaction
+from django.db.models import Model
+from django.core.exceptions import ObjectDoesNotExist
+from django.db.models.query import QuerySet
 from django.utils import timezone
 from django.utils.functional import classproperty
 
@@ -283,6 +286,62 @@ def as_form(self, data=None, files=None, initial=None):
 
         return form
 
+    @staticmethod
+    def serialize_data(data):
+        """
+        This method parses input data (from JobForm usually) and returns a dict which is safe to serialize
+
+        Here we convert the QuerySet of a MultiObjectVar to a list of the pk's and the model instance
+        of an ObjectVar into the pk value.
+
+        These are converted back during job execution.
+        """
+        return_data = {}
+        for field_name, value in data.items():
+            if isinstance(value, QuerySet):
+                return_data[field_name] = list(value.values_list("pk", flat=True))
+            elif isinstance(value, Model):
+                return_data[field_name] = value.pk
+            else:
+                return_data[field_name] = value
+
+        return return_data
+
+    @classmethod
+    def deserialize_data(cls, data):
+        """
+        Given data input for a job execution, deserialize it by resolving object references using defined variables.
+
+        This converts a list of pk's back into a QuerySet for MultiObjectVar instances and single pk values into
+        model instances for ObjectVar.
+
+        Note that when resolving querysets or model instances by their PK, we do not catch DoesNotExist
+        exceptions here, we leave it up the caller to handle those cases. The normal job execution code
+        path would consider this a failure of the job execution, as described in `nautobot.extras.jobs.run_job`.
+        """
+        vars = cls._get_vars()
+        return_data = {}
+
+        for field_name, value in data.items():
+            var = vars[field_name]
+            if isinstance(var, MultiObjectVar):
+                queryset = var.field_attrs["queryset"].filter(pk__in=value)
+                if queryset.count() < len(value):
+                    # Not all objects found
+                    not_found_pk_list = value - list(queryset.values_list("pk", flat=True))
+                    raise queryset.model.DoesNotExist(
+                        f"Failed to find requested objects for var {field_name}: [{', '.join(not_found_pk_list)}]"
+                    )
+                return_data[field_name] = var.field_attrs["queryset"].filter(pk__in=value)
+
+            elif isinstance(var, ObjectVar):
+                return_data[field_name] = var.field_attrs["queryset"].get(pk=value)
+
+            else:
+                return_data[field_name] = value
+
+        return return_data
+
     def run(self, data, commit):
         """
         Method invoked when this Job is run, before any "test_*" methods.
@@ -816,6 +875,19 @@ def run_job(data, request, job_result_pk, commit=True, *args, **kwargs):
     job = job_class()
     job.job_result = job_result
 
+    # Attempt to resolve serialized data back into original form by creating querysets or model instances
+    # If we fail to find any objects, we consider this a job execution error, and fail.
+    # This might happen when a job sits on the queue for a while (i.e. scheduled) and data has changed
+    # or it might be bad input from an API request, or manual execution.
+    try:
+        data = job_class.deserialize_data(data)
+    except ObjectDoesNotExist as e:
+        job_result.set_status(JobResultStatusChoices.STATUS_ERRORED)
+        job.log_failure(message=e)
+        job_result.completed = timezone.now()
+        job_result.save()
+        return False
+
     if job.read_only:
         # Force commit to false for read only jobs.
         commit = False
@@ -824,7 +896,7 @@ def run_job(data, request, job_result_pk, commit=True, *args, **kwargs):
 
     job.logger.info(f"Running job (commit={commit})")
 
-    job_result.status = JobResultStatusChoices.STATUS_RUNNING
+    job_result.set_status(JobResultStatusChoices.STATUS_RUNNING)
     job_result.save()
 
     # Add any files to the form data
diff --git a/nautobot/extras/views.py b/nautobot/extras/views.py
--- a/nautobot/extras/views.py
+++ b/nautobot/extras/views.py
@@ -707,7 +707,7 @@ def post(self, request, class_path):
                 job.class_path,
                 job_content_type,
                 request.user,
-                data=form.cleaned_data,
+                data=job_class.serialize_data(form.cleaned_data),
                 request=copy_safe_request(request),
                 commit=commit,
             )


Test patch: