Repository: netbox-community/netbox
Pull request: Closes #16971: Add system jobs
<!--
    Thank you for your interest in contributing to NetBox! Please note that
    our contribution policy requires that a feature request or bug report be
    approved and assigned prior to opening a pull request. This helps avoid
    waste time and effort on a proposed change that we might not be able to
    accept.

    IF YOUR PULL REQUEST DOES NOT REFERENCE AN ISSUE WHICH HAS BEEN ASSIGNED
    TO YOU, IT WILL BE CLOSED AUTOMATICALLY.

    Please specify your assigned issue number on the line below.
-->
### Fixes: #16971

<!--
    Please include a summary of the proposed changes below.
-->

This PR allows the NetBox core and plugins to register job runners as system jobs. These are automatically scheduled without UI interaction, just before the worker process starts. This allows job runners to be used for housekeeping and synchronization tasks in future commits.

Problem statement:
Allow plugin authors to register housekeeping tasks
### NetBox version

v4.0.7

### Feature type

New functionality

### Proposed functionality

NetBox includes a housekeeping management command that _usually_ runs nightly. This command handles:

- Clearing expired authentication sessions from the database
- Deleting older changelog records
- Deleting older job result records
- Checking for new NetBox releases

It would be great if plugin authors could register additional tasks to run by the housekeeping process without having to write their own management commands and setup mechanisms to run those periodically.

### Use case

I often have to write periodic tasks to alter model instances in my plugins, reacting to dates, sending emails etc. and having to setup the whole management commands and related mechanisms to run those periodically is not really convenient, even more so considering that NetBox already has mechanisms built in to do so.

### Database changes

None whatsoever

### External dependencies

None

Hints from issue discussion:
The background jobs framework of #15692 will include the ability to schedule system jobs, i.e. housekeeping or other tasks that run periodically in the background. While the plugin still requires a special management command for interactive execution (optional), periodic execution is covered by this framework.
I think ultimately we probably want to do away with the separate housekeeping task and integrate everything using the job's framework, particularly in light of @alehaa's recent work.
The `JobRunner` framework now supports the implementation of any kind of task, including housekeeping. However, the automatic registration of jobs on plugin load did not make it into the PR because the implementation is not stable enough for production. I suggest this FR to focus on this remaining issue, so that even housekeeping tasks can be set up automatically and solve this problem.

Here are the latest findings:

* Calling `JobRunner.enqueue_once()` directly from the `ready()` method of `AppConfig` logs a warning because the database is accessed during plugin setup. The [Django documentation](https://docs.djangoproject.com/en/5.0/ref/applications/#django.apps.AppConfig.ready) explicitly warns against this use case, as the `ready()` method is called every time a plugin is loaded, even if it is only for the `nbshell` management command.
* Using the [`connection_created`](https://docs.djangoproject.com/en/5.0/ref/signals/#connection-created) signal solves the console warning. However, this triggers the `JobRunner` setup even more often, e.g. because the database is connected before each job execution.

So the problem for now is to find a way to automatically trigger the setup method outside of the request-response cycle that is called only once or say every few hours.
How about adding the auto-registration to the beginning of the `rqworker` management command? It would only be called once for each worker started - not for the frontend and not for each database connection. After updates, restarting the `rqworker` would trigger auto-registration again.

I'd implement this using a new method in `AppConfig` (e.g. `enqueue_job_runner()`) being called by `rqworker.handle()` for each plugin. An alternative would be using a signal, which avoids a new method but could obfuscate the logic.
> How about adding the auto-registration to the beginning of the `rqworker` management command? It would only be called once for each worker started - not for the frontend and not for each database connection. After updates, restarting the `rqworker` would trigger auto-registration again.
> 
> I'd implement this using a new method in `AppConfig` (e.g. `enqueue_job_runner()`) being called by `rqworker.handle()` for each plugin. An alternative would be using a signal, which avoids a new method but could obfuscate the logic.

This feels like a decent approach. 
Netbox also has existing mechanisms working with "registries"; we could have a "Housekeeping tasks" registry which would be handled in a similar fashion by the worker and would provide a standardized way to register periodic tasks. 
I'm not sure if this is possible. Usually the registry handles static variables like a list of middleware to load. To register a `JobRunner`, its `enqueue_once()` method needs to be called and its arguments can vary depending on the plugin's implementation. Therefore I think some kind of method needs to be called for each plugin, but of course it can reside in the `AppConfig` class along with other registry-related variables. In this method, plugins could implement the logic to setup the `JobRunner`, i.e. by loading configuration or scheduling task depending on other NetBox objects, etc.
@jeremystretch can you assign this to me so I can work on a PR? Maybe we can still make it into 4.1.
@jeremystretch did you have any chance to review this? This would be the last missing piece to add auto-scheduling background tasks, which didn't make it into the `JobRunner` framework.
@alehaa I am not so sure about the approach, I think something more agnostic that can be called **after** initialization of all apps are done but not specifically tied jobs or RQ might be a better approach.

That said, I don't know if there is a viable solution for this as there is no "apps_ready" signal emitted by django.  Your solution might be the most viable right now.

Commit message: Add predefined job interval choices

Patch to review:
diff --git a/docs/plugins/development/background-jobs.md b/docs/plugins/development/background-jobs.md
index 873390a5886..c1a313704be 100644
--- a/docs/plugins/development/background-jobs.md
+++ b/docs/plugins/development/background-jobs.md
@@ -29,14 +29,25 @@ class MyTestJob(JobRunner):
 
 You can schedule the background job from within your code (e.g. from a model's `save()` method or a view) by calling `MyTestJob.enqueue()`. This method passes through all arguments to `Job.enqueue()`. However, no `name` argument must be passed, as the background job name will be used instead.
 
+!!! tip
+    A set of predefined intervals can be used from `core.choices.JobIntervalChoices`.
+
 ### Attributes
 
-`JobRunner` attributes are defined under a class named `Meta` within the job. These are optional, but encouraged.
+`JobRunner` attributes are defined under a class named `Meta` within the job. These are optional (unless specified otherwise), but encouraged.
 
 #### `name`
 
 This is the human-friendly names of your background job. If omitted, the class name will be used.
 
+#### `system_enabled`
+
+When the `JobRunner` is defined as [system job](#system-jobs), this attribute controls whether a job will be scheduled. By default, this attribute is `True`.
+
+#### `system_interval` *(required for system jobs)*
+
+When the `JobRunner` is defined as [system job](#system-jobs), this attribute controls the interval of the scheduled job.
+
 ### Scheduled Jobs
 
 As described above, jobs can be scheduled for immediate execution or at any later time using the `enqueue()` method. However, for management purposes, the `enqueue_once()` method allows a job to be scheduled exactly once avoiding duplicates. If a job is already scheduled for a particular instance, a second one won't be scheduled, respecting thread safety. An example use case would be to schedule a periodic task that is bound to an instance in general, but not to any event of that instance (such as updates). The parameters of the `enqueue_once()` method are identical to those of `enqueue()`.
@@ -46,25 +57,49 @@ As described above, jobs can be scheduled for immediate execution or at any late
 
 #### Example
 
+```python title="models.py"
+from django.db import models
+from core.choices import JobIntervalChoices
+from netbox.models import NetBoxModel
+from .jobs import MyTestJob
+
+class MyModel(NetBoxModel):
+    foo = models.CharField()
+
+    def save(self, *args, **kwargs):
+        MyTestJob.enqueue_once(instance=self, interval=JobIntervalChoices.INTERVAL_HOURLY)
+        return super().save(*args, **kwargs)
+
+    def sync(self):
+        MyTestJob.enqueue(instance=self)
+```
+
+
+### System Jobs
+
+Some plugins may implement background jobs that are decoupled from any object and the request-response cycle. Typical use cases would be housekeeping tasks or synchronization jobs. These can be created using *system jobs*. The `JobRunner` class has everything included to provide this type of job as well. Just add the appropriate metadata to let NetBox schedule all background jobs automatically.
+
+!!! info
+    All system jobs are automatically scheduled just before the `./manage.py rqworker` command is started and the job queue is processed. The schedules are also checked at each restart of this process.
+
+#### Example
+
 ```python title="jobs.py"
+from core.choices import JobIntervalChoices
 from netbox.jobs import JobRunner
-
+from .models import MyModel
 
 class MyHousekeepingJob(JobRunner):
     class Meta:
-        name = "Housekeeping"
+        name = "My Housekeeping Job"
+        system_interval = JobIntervalChoices.INTERVAL_HOURLY  # or integer for n minutes
 
     def run(self, *args, **kwargs):
-        # your logic goes here
-```
+        MyModel.objects.filter(foo='bar').delete()
 
-```python title="__init__.py"
-from netbox.plugins import PluginConfig
-
-class MyPluginConfig(PluginConfig):
-    def ready(self):
-        from .jobs import MyHousekeepingJob
-        MyHousekeepingJob.setup(interval=60)
+system_jobs = (
+    MyHousekeepingJob,
+)
 ```
 
 ## Task queues
diff --git a/docs/plugins/development/data-backends.md b/docs/plugins/development/data-backends.md
index 8b7226a4139..0c6d44d9516 100644
--- a/docs/plugins/development/data-backends.md
+++ b/docs/plugins/development/data-backends.md
@@ -18,6 +18,6 @@ backends = [MyDataBackend]
 ```
 
 !!! tip
-    The path to the list of search indexes can be modified by setting `data_backends` in the PluginConfig instance.
+    The path to the list of data backends can be modified by setting `data_backends` in the PluginConfig instance.
 
 ::: netbox.data_backends.DataBackend
diff --git a/netbox/core/choices.py b/netbox/core/choices.py
index 01a072ce1cf..442acc26bac 100644
--- a/netbox/core/choices.py
+++ b/netbox/core/choices.py
@@ -72,6 +72,20 @@ class JobStatusChoices(ChoiceSet):
     )
 
 
+class JobIntervalChoices(ChoiceSet):
+    INTERVAL_MINUTELY = 1
+    INTERVAL_HOURLY = 60
+    INTERVAL_DAILY = 60 * 24
+    INTERVAL_WEEKLY = 60 * 24 * 7
+
+    CHOICES = (
+        (INTERVAL_MINUTELY, _('Minutely')),
+        (INTERVAL_HOURLY, _('Hourly')),
+        (INTERVAL_DAILY, _('Daily')),
+        (INTERVAL_WEEKLY, _('Weekly')),
+    )
+
+
 #
 # ObjectChanges
 #
diff --git a/netbox/core/management/commands/rqworker.py b/netbox/core/management/commands/rqworker.py
index e1fb6fd1118..7815c7d569f 100644
--- a/netbox/core/management/commands/rqworker.py
+++ b/netbox/core/management/commands/rqworker.py
@@ -1,7 +1,10 @@
 import logging
 
+from django.core.management.base import CommandError
 from django_rq.management.commands.rqworker import Command as _Command
 
+from netbox.registry import registry
+
 
 DEFAULT_QUEUES = ('high', 'default', 'low')
 
@@ -14,6 +17,16 @@ class Command(_Command):
     of only the 'default' queue).
     """
     def handle(self, *args, **options):
+        # Setup system jobs.
+        for job in registry['system_jobs'].values():
+            if getattr(job.Meta, 'system_enabled', True):
+                try:
+                    logger.debug(f"Scheduling system job {job.name}")
+                    job.enqueue_once(interval=getattr(job.Meta, 'system_interval'))
+
+                except AttributeError as e:
+                    raise CommandError(f"Job {job.name} is missing required attribute in Meta: {e.name}")
+
         # Run the worker with scheduler functionality
         options['with_scheduler'] = True
 
diff --git a/netbox/netbox/jobs.py b/netbox/netbox/jobs.py
index 087c2489687..b357c714473 100644
--- a/netbox/netbox/jobs.py
+++ b/netbox/netbox/jobs.py
@@ -127,7 +127,7 @@ class scheduled for `instance`, the existing job will be updated if necessary. T
         if job:
             # If the job parameters haven't changed, don't schedule a new job and keep the current schedule. Otherwise,
             # delete the existing job and schedule a new job instead.
-            if (schedule_at and job.scheduled == schedule_at) and (job.interval == interval):
+            if (not schedule_at or job.scheduled == schedule_at) and (job.interval == interval):
                 return job
             job.delete()
 
diff --git a/netbox/netbox/plugins/__init__.py b/netbox/netbox/plugins/__init__.py
index e2f0f22fce4..76bb9d26bcc 100644
--- a/netbox/netbox/plugins/__init__.py
+++ b/netbox/netbox/plugins/__init__.py
@@ -8,7 +8,7 @@
 
 from netbox.registry import registry
 from netbox.search import register_search
-from netbox.utils import register_data_backend
+from netbox.utils import register_data_backend, register_system_job
 from .navigation import *
 from .registration import *
 from .templates import *
@@ -26,6 +26,7 @@
 DEFAULT_RESOURCE_PATHS = {
     'search_indexes': 'search.indexes',
     'data_backends': 'data_backends.backends',
+    'system_jobs': 'jobs.system_jobs',
     'graphql_schema': 'graphql.schema',
     'menu': 'navigation.menu',
     'menu_items': 'navigation.menu_items',
@@ -73,6 +74,7 @@ class PluginConfig(AppConfig):
     # Optional plugin resources
     search_indexes = None
     data_backends = None
+    system_jobs = None
     graphql_schema = None
     menu = None
     menu_items = None
@@ -111,6 +113,11 @@ def ready(self):
         for backend in data_backends:
             register_data_backend()(backend)
 
+        # Register system jobs (if defined)
+        system_jobs = self._load_resource('system_jobs') or []
+        for job in system_jobs:
+            register_system_job()(job)
+
         # Register template content (if defined)
         if template_extensions := self._load_resource('template_extensions'):
             register_template_extensions(template_extensions)
diff --git a/netbox/netbox/registry.py b/netbox/netbox/registry.py
index 0920cbccf96..48d7921f259 100644
--- a/netbox/netbox/registry.py
+++ b/netbox/netbox/registry.py
@@ -30,6 +30,7 @@ def __delitem__(self, key):
     'models': collections.defaultdict(set),
     'plugins': dict(),
     'search': dict(),
+    'system_jobs': dict(),
     'tables': collections.defaultdict(dict),
     'views': collections.defaultdict(dict),
     'widgets': dict(),
diff --git a/netbox/netbox/tests/dummy_plugin/jobs.py b/netbox/netbox/tests/dummy_plugin/jobs.py
new file mode 100644
index 00000000000..bf6560fd6ed
--- /dev/null
+++ b/netbox/netbox/tests/dummy_plugin/jobs.py
@@ -0,0 +1,15 @@
+from core.choices import JobIntervalChoices
+from netbox.jobs import JobRunner
+
+
+class DummySystemJob(JobRunner):
+    class Meta:
+        system_interval = JobIntervalChoices.INTERVAL_HOURLY
+
+    def run(self, *args, **kwargs):
+        pass
+
+
+system_jobs = (
+    DummySystemJob,
+)
diff --git a/netbox/netbox/tests/test_jobs.py b/netbox/netbox/tests/test_jobs.py
index cb3024038ae..0e230d46772 100644
--- a/netbox/netbox/tests/test_jobs.py
+++ b/netbox/netbox/tests/test_jobs.py
@@ -90,6 +90,15 @@ def test_enqueue_once_twice_same(self):
         self.assertEqual(job1, job2)
         self.assertEqual(TestJobRunner.get_jobs(instance).count(), 1)
 
+    def test_enqueue_once_twice_same_no_schedule_at(self):
+        instance = Job()
+        schedule_at = self.get_schedule_at()
+        job1 = TestJobRunner.enqueue_once(instance, schedule_at=schedule_at)
+        job2 = TestJobRunner.enqueue_once(instance)
+
+        self.assertEqual(job1, job2)
+        self.assertEqual(TestJobRunner.get_jobs(instance).count(), 1)
+
     def test_enqueue_once_twice_different_schedule_at(self):
         instance = Job()
         job1 = TestJobRunner.enqueue_once(instance, schedule_at=self.get_schedule_at())
@@ -127,3 +136,30 @@ def test_enqueue_once_after_enqueue(self):
         self.assertNotEqual(job1, job2)
         self.assertRaises(Job.DoesNotExist, job1.refresh_from_db)
         self.assertEqual(TestJobRunner.get_jobs(instance).count(), 1)
+
+
+class SystemJobTest(JobRunnerTestCase):
+    """
+    Test that system jobs can be scheduled.
+
+    General functionality already tested by `JobRunnerTest` and `EnqueueTest`.
+    """
+
+    def test_scheduling(self):
+        # Can job be enqueued?
+        job = TestJobRunner.enqueue(schedule_at=self.get_schedule_at())
+        self.assertIsInstance(job, Job)
+        self.assertEqual(TestJobRunner.get_jobs().count(), 1)
+
+        # Can job be deleted again?
+        job.delete()
+        self.assertRaises(Job.DoesNotExist, job.refresh_from_db)
+        self.assertEqual(TestJobRunner.get_jobs().count(), 0)
+
+    def test_enqueue_once(self):
+        schedule_at = self.get_schedule_at()
+        job1 = TestJobRunner.enqueue_once(schedule_at=schedule_at)
+        job2 = TestJobRunner.enqueue_once(schedule_at=schedule_at)
+
+        self.assertEqual(job1, job2)
+        self.assertEqual(TestJobRunner.get_jobs().count(), 1)
diff --git a/netbox/netbox/tests/test_plugins.py b/netbox/netbox/tests/test_plugins.py
index ba44378c5ce..0fe8549a85a 100644
--- a/netbox/netbox/tests/test_plugins.py
+++ b/netbox/netbox/tests/test_plugins.py
@@ -7,6 +7,7 @@
 
 from netbox.tests.dummy_plugin import config as dummy_config
 from netbox.tests.dummy_plugin.data_backends import DummyBackend
+from netbox.tests.dummy_plugin.jobs import DummySystemJob
 from netbox.plugins.navigation import PluginMenu
 from netbox.plugins.utils import get_plugin_config
 from netbox.graphql.schema import Query
@@ -130,6 +131,13 @@ def test_data_backends(self):
         self.assertIn('dummy', registry['data_backends'])
         self.assertIs(registry['data_backends']['dummy'], DummyBackend)
 
+    def test_system_jobs(self):
+        """
+        Check registered system jobs.
+        """
+        self.assertIn(DummySystemJob.name, registry['system_jobs'])
+        self.assertIs(registry['system_jobs'][DummySystemJob.name], DummySystemJob)
+
     def test_queues(self):
         """
         Check that plugin queues are registered with the accurate name.
diff --git a/netbox/netbox/utils.py b/netbox/netbox/utils.py
index f27d1b5f7fc..60067c55343 100644
--- a/netbox/netbox/utils.py
+++ b/netbox/netbox/utils.py
@@ -3,6 +3,7 @@
 __all__ = (
     'get_data_backend_choices',
     'register_data_backend',
+    'register_system_job',
 )
 
 
@@ -24,3 +25,14 @@ def _wrapper(cls):
         return cls
 
     return _wrapper
+
+
+def register_system_job():
+    """
+    Decorator for registering a `JobRunner` class as system background job.
+    """
+    def _wrapper(cls):
+        registry['system_jobs'][cls.name] = cls
+        return cls
+
+    return _wrapper


Code review comments:
diff_hunk: @@ -24,3 +25,14 @@ def _wrapper(cls):
         return cls
 
     return _wrapper
+
+
+def register_system_job():
+    """
+    Decorator for registering a `JobRunner` class as system background job.
+    """
+    def _wrapper(cls):
+        registry['system_jobs'][cls.name] = cls
path: netbox/netbox/utils.py
text: @user1:
We probably want to namespace these by app or Python path. Otherwise, we risk a name collision among jobs from different plugins.

diff_hunk: @@ -14,6 +17,16 @@ class Command(_Command):
     of only the 'default' queue).
     """
     def handle(self, *args, **options):
+        # Setup system jobs.
+        for job in registry['system_jobs'].values():
+            if getattr(job.Meta, 'system_enabled', True):
path: netbox/core/management/commands/rqworker.py
text: @user1:
Could we do away with the `system_enabled` attribute and just check whether `system_interval` is defined? The former seems redundant since we're already registering system jobs automatically.

diff_hunk: @@ -24,3 +25,14 @@ def _wrapper(cls):
         return cls
 
     return _wrapper
+
+
+def register_system_job():
path: netbox/netbox/utils.py
text: @user1:
What do you think about passing the interval (and potentially other parameters) via the `register_system_job()` wrapper instead of defining them inside the class? They would need to be stored in the registry alongside a reference to the class itself, but it might allow for more flexible reuse of jobs. You could do something like:

```python
register_system_job(interval=HOURLY, do_big_stuff=False)(MyHousekeepingJob)
register_system_job(interval=DAILY, do_big_stuff=True)(MyHousekeepingJob)
```

Just a thought; I might also be overthinking this. Happy to merge as-is.

@author:
I can see the potential in this approach. Would it be okay if the decorator is the only way to register a system job even for plugins instead of listing them in the `system_jobs` variable at the end of the file? `@user2` uses the same approach, but other APIs like data source registration do not allow direct access to the decorator.

@user1:
I think personally I prefer the decorator approach but I don't have a strong opinion. It would probably make sense to encourage the direct use of the decorator for data backend registration in plugins as well. @user3 @user4 @user5 do any of you have thoughts here?

@user1:
We discussed this today and opted to go with the decorator approach as it feels "cleaner" and offers more flexibility.