Repository: nautobot/nautobot

Issue:
Jobs in error stays in Pending state 
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.9
* Nautobot version: 1.1.0

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Start an early failing job as in https://github.com/nautobot/nautobot/issues/732

<!-- What did you expect to happen? -->
### Expected Behavior
Error is reported in the UI, job state's changes to `Errored`

<!-- What happened instead? -->
### Observed Behavior
Job state's stays in `Pending`

### Additional informations

Log example of errored job still displayed as pending:
```
celery_worker_1  | [2021-07-23 18:54:10,983: INFO/MainProcess] Task nautobot.extras.jobs.run_job[e43f026b-1361-4879-9906-19593fcc10f6] received
celery_worker_1  | 18:54:11.002 INFO    nautobot.jobs :
celery_worker_1  |   Repository successfully refreshed
celery_worker_1  | [2021-07-23 18:54:11,002: INFO/ForkPoolWorker-3] Repository successfully refreshed
celery_worker_1  | [2021-07-23 18:54:11,016: ERROR/ForkPoolWorker-3] Task nautobot.extras.jobs.run_job[e43f026b-1361-4879-9906-19593fcc10f6] raised unexpected: AttributeError("'str' object has no attribute 'items'")
celery_worker_1  | Traceback (most recent call last):
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 450, in trace_task
celery_worker_1  |     R = retval = fun(*args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 731, in __protected_call__
celery_worker_1  |     return self.run(*args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/nautobot/extras/jobs.py", line 961, in run_job
celery_worker_1  |     data = job_class.deserialize_data(data)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/nautobot/extras/jobs.py", line 345, in deserialize_data
celery_worker_1  |     for field_name, value in data.items():
celery_worker_1  | AttributeError: 'str' object has no attribute 'items'
```

```
celery_worker_1  | [2021-07-23 15:31:08,187: ERROR/ForkPoolWorker-2] Task nautobot.extras.jobs.run_job[c34588d4-3957-49db-b019-759a4adf72d4] raised unexpected: TypeError('the JSON object must be str, bytes or bytearray, not int')
celery_worker_1  | Traceback (most recent call last):
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 450, in trace_task
celery_worker_1  |     R = retval = fun(*args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 731, in __protected_call__
celery_worker_1  |     return self.run(*args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/nautobot/extras/jobs.py", line 934, in run_job
celery_worker_1  |     job_result = JobResult.objects.get(pk=job_result_pk)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
celery_worker_1  |     return getattr(self.get_queryset(), name)(*args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/cacheops/query.py", line 353, in get
celery_worker_1  |     return qs._no_monkey.get(qs, *args, **kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 425, in get
celery_worker_1  |     num = len(clone)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 269, in __len__
celery_worker_1  |     self._fetch_all()
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/cacheops/query.py", line 283, in _fetch_all
celery_worker_1  |     self._result_cache = list(self._iterable_class(self))
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 70, in __iter__
celery_worker_1  |     for row in compiler.results_iter(results):
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1100, in apply_converters
celery_worker_1  |     value = converter(value, expression, connection)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/json.py", line 83, in from_db_value
celery_worker_1  |     return json.loads(value, cls=self.decoder)
celery_worker_1  |   File "/usr/local/lib/python3.9/json/__init__.py", line 339, in loads
celery_worker_1  |     raise TypeError(f'the JSON object must be str, bytes or bytearray, '
celery_worker_1  | TypeError: the JSON object must be str, bytes or bytearray, not int
```

Please note that I don't know how to reproduce the latter, but would you like that I fill a but with this log?


Discussion:
Thanks for another detailed report @u1735067! This is a pretty straightforward fix. :)

Resolving patch:
diff --git a/nautobot/extras/jobs.py b/nautobot/extras/jobs.py
--- a/nautobot/extras/jobs.py
+++ b/nautobot/extras/jobs.py
@@ -18,7 +18,7 @@
 from django.core.validators import RegexValidator
 from django.db import transaction
 from django.db.models import Model
-from django.core.exceptions import ObjectDoesNotExist
+from django.core.exceptions import NON_FIELD_ERRORS, ObjectDoesNotExist
 from django.db.models.query import QuerySet
 from django.forms import ValidationError
 from django.utils import timezone
@@ -347,6 +347,9 @@ def deserialize_data(cls, data):
         vars = cls._get_vars()
         return_data = {}
 
+        if not isinstance(data, dict):
+            raise TypeError("Data should be a dictionary.")
+
         for field_name, value in data.items():
             # If a field isn't a var, skip it (e.g. `_commit`).
             try:
@@ -965,7 +968,14 @@ def run_job(data, request, job_result_pk, commit=True, *args, **kwargs):
     """
     from nautobot.extras.models import JobResult  # avoid circular import
 
-    job_result = JobResult.objects.get(pk=job_result_pk)
+    # Getting the correct job result can fail if the stored data cannot be serialized.
+    # Catching `TypeError: the JSON object must be str, bytes or bytearray, not int`
+    try:
+        job_result = JobResult.objects.get(pk=job_result_pk)
+    except TypeError as e:
+        logger.error(f"Unable to serialize data for job {job_result_pk}")
+        logger.error(e)
+        return False
 
     job_class = get_job(job_result.name)
     if not job_class:
@@ -979,26 +989,46 @@ def run_job(data, request, job_result_pk, commit=True, *args, **kwargs):
         job_result.completed = timezone.now()
         job_result.save()
         return False
-    job = job_class()
-    job.job_result = job_result
-
-    # Capture the file IDs for any FileProxy objects created so we can cleanup later.
-    file_fields = list(job._get_file_vars())
-    file_ids = [data[f] for f in file_fields]
 
-    # Attempt to resolve serialized data back into original form by creating querysets or model instances
-    # If we fail to find any objects, we consider this a job execution error, and fail.
-    # This might happen when a job sits on the queue for a while (i.e. scheduled) and data has changed
-    # or it might be bad input from an API request, or manual execution.
+    try:
+        job = job_class()
+        job.active_test = "initialization"
+        job.job_result = job_result
+    except Exception as error:
+        logger.error("Error initializing job object.")
+        logger.error(error)
+        stacktrace = traceback.format_exc()
+        job_result.log_failure(f"Error initializing job:\n```\n{stacktrace}\n```")
+        job_result.set_status(JobResultStatusChoices.STATUS_ERRORED)
+        job_result.completed = timezone.now()
+        job_result.save()
+        return False
 
     try:
+        # Capture the file IDs for any FileProxy objects created so we can cleanup later.
+        file_ids = None
+        file_fields = list(job._get_file_vars())
+        file_ids = [data[f] for f in file_fields]
+
+        # Attempt to resolve serialized data back into original form by creating querysets or model instances
+        # If we fail to find any objects, we consider this a job execution error, and fail.
+        # This might happen when a job sits on the queue for a while (i.e. scheduled) and data has changed
+        # or it might be bad input from an API request, or manual execution.
+
         data = job_class.deserialize_data(data)
-    except ObjectDoesNotExist as e:
+    except Exception:
         job_result.set_status(JobResultStatusChoices.STATUS_ERRORED)
-        job.log_failure(message=e)
+        stacktrace = traceback.format_exc()
+        job_result.log(
+            f"Error initializing job:\n```\n{stacktrace}\n```",
+            level_choice=LogLevelChoices.LOG_FAILURE,
+            grouping="initialization",
+            logger=logger,
+        )
         job_result.completed = timezone.now()
         job_result.save()
-        job.delete_files(*file_ids)  # Cleanup FileProxy objects
+        if file_ids:
+            job.delete_files(*file_ids)  # Cleanup FileProxy objects
         return False
 
     if job.read_only:


Test patch:
diff --git a/nautobot/extras/tests/test_jobs.py b/nautobot/extras/tests/test_jobs.py
--- a/nautobot/extras/tests/test_jobs.py
+++ b/nautobot/extras/tests/test_jobs.py
@@ -260,6 +260,36 @@ def test_object_vars(self):
             self.assertEqual(job_result.status, JobResultStatusChoices.STATUS_COMPLETED)
             self.assertEqual({"role": str(d.pk), "roles": [str(d.pk)]}, job_result_data)
 
+    def test_job_data_as_string(self):
+        """
+        Test that job doesn't error when not a dictionary.
+        """
+        with self.settings(JOBS_ROOT=os.path.join(settings.BASE_DIR, "extras/tests/dummy_jobs")):
+            module = "test_object_vars"
+            name = "TestObjectVars"
+            job_class = get_job(f"local/{module}/{name}")
+
+            # Prepare the job data
+            job_result = JobResult.objects.create(
+                name=job_class.class_path,
+                obj_type=self.job_content_type,
+                user=None,
+                job_id=uuid.uuid4(),
+            )
+            data = "BAD DATA STRING"
+            run_job(data=data, request=None, commit=False, job_result_pk=job_result.pk)
+            job_result.refresh_from_db()
+
+            # Assert stuff
+            self.assertEqual(job_result.status, JobResultStatusChoices.STATUS_ERRORED)
+            self.assertIn(
+                "Data should be a dictionary",
+                next(
+                    log[-1]  # actual log message in the logging tuple
+                    for log in job_result.data["initialization"]["log"]
+                ),
+            )
+
 
 class JobFileUploadTest(TestCase):
     """Test a job that uploads/deletes files."""