Repository: netbox-community/netbox

Issue:
Allow plugin to create their own django-rq queues
### NetBox version

v2.11.7

### Feature type

New functionality

### Proposed functionality

I'd like to propose the ability for netbox plugin to define some dedicated _django-rq_ queue(s). This new feature has already been discussed in #4908 but was missing a owner back in time.

The idea would be to store queue names in a new attribute inside the `PluginConfig` class. This attributes would be a `list` in order to defined several queue to create by django-rq.

In order to avoid queue name collision, the actual queue names will be prepended by the plugin name.

As an exemple the following config will lead to 3 queues being created using django-rq:

* _my_super_plugin.high_
* _my_super_plugin.medium_
* _my_super_plugin.low_

```python
MyPluginConfig(PluginConfig):
    name = "my_super_plugin"
    queues = [
        "low"
        "medium"
        "high"
    ]
```

The creation of the queues will need to override django settings ([RQ_QUEUES](https://github.com/netbox-community/netbox/blob/607039f043f0a43437db05aae18bae6f1ef49f85/netbox/netbox/settings.py#L571)) and can be done in the `ready` method of the `PluginConfig` class.

We can also override django RQ_QUEUES settings directly in the `settings.py`, during [init of all plugins](https://github.com/netbox-community/netbox/blob/4292d88a9230789ef0c5af627e7245d2d3dd4bda/netbox/netbox/settings.py#L597).

If you are interested in such feature I can take care of a PR.

### Use case

All netbox plugins have the possibility to use the _default_ queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

### Database changes

_No response_

### External dependencies

_No response_


Discussion:
I already added this feature in [my own netbox fork](https://github.com/maximumG/netbox/commit/9a0bc730a3b3b60098a57fd0a0e180d19983ecc2) along with some new unittest functions to ensure the implementation is working as expected

If we agree on this I can move on and update the netbox documentation to add some notes regarding dedicated plugin queues.
Thanks for this!

> All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that `rqworker` services each queue respectively. We need a way to order queues among both NetBox core and one or more plugins in a predictable manner. For example, suppose two different plugins each introduce their own high- and low-priority queues. We don't want to end up ordering them as:

```
[default, plugin1-high, plugin1-low, plugin2-high, plugin2-low]
```

Without such a mechanism in place, plugins would be relying on administrators to manually reconfigure their `rqworker` invocations accordingly, which is sure to lead to frustration. I'm open to suggestions.

NetBox also currently has a `check_releases` queue that is used solely for checking for new releases. I don't recall the motivation for this design decision initially, but it's probably best moved to a scheduled task within the default queue using [`rq-scheduler`](https://github.com/rq/rq-scheduler) anyway. (In fact, this may become necessary if we decided to remove caching support as discussed under #6639.)

> Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively.

You are absolutely right, I forgot to mention half of the proposed solution...sorry for this 😞 

> We need a way to order queues among both NetBox core and one or more plugins in a predictable manner.

From my point of view, queue administration should be tackled at the sysadmin level. What I had in mind in the first place is to have:

* per default, an rqworker instance listening to queue `default` and `check_release`, to handle netbox core jobs (already the case now)
* a new rqworker instance configured per plugin. This new instance must be configured to listen only to the plugin's queues

The second point implies that the plugin maintainer inform potential user that they must create a dedicated rqworker instance listening to the plugin queues (container / systemd unit). No matter the underlying implementation to spawn new rqworker it is really simple nowadays to add an additional container/pod (cloud) or systemd unit (on-premise) to handle each plugin background workload.

Even if creating new rqworker instance is a manual operation done by sysadmin, I see the following benefits:

- Each plugin background workload is isolated and does not impact other plugin or netbox core workload
- Netbox source code is "kept clean" and focused on IRM only, as we do not add some code to handle queueing system
- We keep it relatively simple by separating concerns of Netbox code and system implementation details
Maybe there's a middle ground, where NetBox provides its own high- and low-priority queues (in addition to the default queue) for consumption by plugins. NetBox's rqworker would service all three of these accordingly by default. Then I'd feel more comfortable putting the burden on the user to service any additional queues needed by plugins.
I will try to sum up our two ideas to define a proposal target design.

### What Netbox would provides out of the box

**Netbox would define 3 queues** : `default`, `high-priority` and `low-priority`. The `default` queue is already present and dedicated for netbox internal-use (reports and scripts jobs). The `high-priority` and `low-priority` queues are provided out of the box to plugin developer if they need background job and do not want to implement their own queues.

**Netbox would provides a single _rq-worker_** (systemd/container) which listen to the 3 queues above. As stated by the [RQ documentation](https://python-rq.org/docs/workers/), the queue order is important here and should stay as `default`, `high-priority` and `low-priority`. This way we ensure `default` job > `high-priority` jobs > `low-priority` jobs

### What Netbox would allow for plugin developers

Plugin developers have the ability to define their own dedicated queue(s) in the plugin's config. This implies a new rq-worker instance must be created and configured to listen to the plugin dedicated queue(s). In such case, end-user of the plugin must ensure this rq-worker is installed and configured properly according to any plugin install documentation.
I like that! Just one tweak:

> the queue order is important here and should stay as default, high-priority and low-priority.

I would arrange them high > default > low. NetBox would continue to queue webhooks, reports, etc. in the default queue.
I pinged the Docker folks for their take on this as well. Per @tobiasge in the #netbox-docker channel on the NetDev Slack:

> In the docker-compose.yaml we have defined a container to run the default rqworker. See this configuration. To define additional workers one would need to add a new worker instance to the docker-compose.override.yml file, which is also explained in our documentation.
> From that point of view, I think it is easy to add new workers for plugins. So no objections from me.
I should also note that the PR for this should be based on the `feature` branch, as it will need to go into the next major release.
Thanks for the update @jeremystretch 

I'd also wanted to confirm where should I put some words in the documentation regarding this new feature (default queue + plugin related) ?

I was thinking about a new dedicated section inside [the plugin page](https://netbox.readthedocs.io/en/stable/plugins/development/) right after the caching configuration.
@maximumG Yep, that sounds perfect!

Resolving patch:
diff --git a/netbox/extras/plugins/__init__.py b/netbox/extras/plugins/__init__.py
--- a/netbox/extras/plugins/__init__.py
+++ b/netbox/extras/plugins/__init__.py
@@ -47,6 +47,9 @@ class PluginConfig(AppConfig):
     # Middleware classes provided by the plugin
     middleware = []
 
+    # Django-rq queues dedicated to the plugin
+    queues = []
+
     # Default integration paths. Plugin authors can override these to customize the paths to
     # integrated components.
     template_extensions = 'template_content.template_extensions'
diff --git a/netbox/netbox/settings.py b/netbox/netbox/settings.py
--- a/netbox/netbox/settings.py
+++ b/netbox/netbox/settings.py
@@ -546,7 +546,9 @@ def _setting(name, default=None):
     }
 
 RQ_QUEUES = {
+    'high': RQ_PARAMS,
     'default': RQ_PARAMS,
+    'low': RQ_PARAMS,
 }
 
 
@@ -599,3 +601,14 @@ def _setting(name, default=None):
     plugin_middleware = plugin_config.middleware
     if plugin_middleware and type(plugin_middleware) in (list, tuple):
         MIDDLEWARE.extend(plugin_middleware)
+
+    # Create RQ queues dedicated to the plugin
+    # we use the plugin name as a prefix for queue name's defined in the plugin config
+    # ex: mysuperplugin.mysuperqueue1
+    if type(plugin_config.queues) is not list:
+        raise ImproperlyConfigured(
+            "Plugin {} queues must be a list.".format(plugin_name)
+        )
+    RQ_QUEUES.update({
+        f"{plugin_name}.{queue}": RQ_PARAMS for queue in plugin_config.queues
+    })


Test patch:
diff --git a/netbox/extras/tests/dummy_plugin/__init__.py b/netbox/extras/tests/dummy_plugin/__init__.py
--- a/netbox/extras/tests/dummy_plugin/__init__.py
+++ b/netbox/extras/tests/dummy_plugin/__init__.py
@@ -12,6 +12,11 @@ class DummyPluginConfig(PluginConfig):
     middleware = [
         'extras.tests.dummy_plugin.middleware.DummyMiddleware'
     ]
+    queues = [
+        'testing-low',
+        'testing-medium',
+        'testing-high'
+    ]
 
 
 config = DummyPluginConfig
diff --git a/netbox/extras/tests/test_plugins.py b/netbox/extras/tests/test_plugins.py
--- a/netbox/extras/tests/test_plugins.py
+++ b/netbox/extras/tests/test_plugins.py
@@ -80,6 +80,14 @@ def test_middleware(self):
         """
         self.assertIn('extras.tests.dummy_plugin.middleware.DummyMiddleware', settings.MIDDLEWARE)
 
+    def test_queues(self):
+        """
+        Check that plugin queues are registered with the accurate name.
+        """
+        self.assertIn('extras.tests.dummy_plugin.testing-low', settings.RQ_QUEUES)
+        self.assertIn('extras.tests.dummy_plugin.testing-medium', settings.RQ_QUEUES)
+        self.assertIn('extras.tests.dummy_plugin.testing-high', settings.RQ_QUEUES)
+
     def test_min_version(self):
         """
         Check enforcement of minimum NetBox version.