Repository: nautobot/nautobot

Issue:
nautobot-scheduler can't apply celery_backend_cleanup_1
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.9
* Nautobot version: 1.2.5

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Run the scheduler as systemd-service
2.
3.

<!-- What did you expect to happen? -->
### Expected Behavior
Internal jobs run without an error

<!-- What happened instead? -->
### Observed Behavior
```
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]: [2022-02-08 04:00:00,000: INFO/MainProcess] Scheduler: Sending due task celery.backend_cleanup_1 (celery.backend_cleanup)
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]: [2022-02-08 04:00:00,001: ERROR/MainProcess] Message Error: Couldn't apply scheduled task celery.backend_cleanup_1: 'str' object has no attribute 'items'
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]: ['  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>\n    sys.exit(main())\n', '  File "/opt/nautobot/lib/python3.9/site-packages/nautobot/cor>
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]: Traceback (most recent call last):
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]:   File "/opt/nautobot/lib/python3.9/site-packages/celery/beat.py", line 402, in apply_async
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]:     entry_kwargs = _evaluate_entry_kwargs(entry.kwargs)
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]:   File "/opt/nautobot/lib/python3.9/site-packages/celery/beat.py", line 220, in _evaluate_entry_kwargs
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]:     for k, v in entry_kwargs.items()
Feb 08 04:00:00 dh01-a-06-18 nautobot-server[3033678]: AttributeError: 'str' object has no attribute 'items'
```

I see this error for every day.


Discussion:
Thanks for the report. I can confirm that this issue is reproducible in a Nautobot development environment by leaving it running overnight.

Resolving patch:
diff --git a/nautobot/core/celery/schedulers.py b/nautobot/core/celery/schedulers.py
--- a/nautobot/core/celery/schedulers.py
+++ b/nautobot/core/celery/schedulers.py
@@ -2,6 +2,7 @@
 
 from celery import current_app
 from django_celery_beat.schedulers import ModelEntry, DatabaseScheduler
+from kombu.utils.json import loads
 
 from nautobot.extras.models import ScheduledJob, ScheduledJobs
 
@@ -20,8 +21,14 @@ def __init__(self, model, app=None):
         self.app = app or current_app._get_current_object()
         self.name = "{}_{}".format(model.name, model.pk)
         self.task = model.task
-        self.args = model.args
-        self.kwargs = model.kwargs
+        try:
+            # Nautobot scheduled jobs pass args/kwargs as constructed objects,
+            # but Celery built-in jobs such as celery.backend_cleanup pass them as JSON to be parsed
+            self.args = model.args if isinstance(model.args, (tuple, list)) else loads(model.args or "[]")
+            self.kwargs = model.kwargs if isinstance(model.kwargs, dict) else loads(model.kwargs or "{}")
+        except (TypeError, ValueError) as exc:
+            logger.exception("Removing schedule %s for argument deserialization error: %s", self.name, exc)
+            self._disable(model)
         try:
             self.schedule = model.schedule
         except model.DoesNotExist:


Test patch: