Package: pluto-rt
Pluto-RT: Real-time web results for long-running processesAny time you have a need to trigger a long-running process from a web view, you run into Python's
blocking nature - nothing will be displayed until the process is complete, and you'll hit timeouts
if you exceed the default timeout for your web application process.To solve the long-running aspect, we turn to background workers like django-q or django-celery. But
then the user has no insight into what the background worker is doing.Pluto-RT solves that by using Redis as a message queuing service with a FIFO (first in, first out)
queue. Messages can be placed on the queue by the background worker and when they are retrieved by
the view, it returns the oldest ones first. It can work by polling with a WSGI server, and using server-sent events (SSE) with ASGI.https://user-images.githubusercontent.com/102694/233860792-652f8790-6f31-4dd8-8c37-fc479171c576.movThe overall strategy is this:Create a unique "queue name" which can be sent to a worker queue and passed into a "results" page.Invoke your background processor (worker) with that queue name. The worker places messages onto the queue as it progresses with the task.Display the results template, passing it the queue name. The template generates htmx which retrieves messages associated with that queue.The server removes the oldest messages from the queue and delivers them to the client.DemoThere is a demo available in the demo directory. The quickest way to try it out is using docker:docker compose --project-directory demo up -dThen open your browser tohttp://localhost:8000/for SSE, andhttp://localhost:8001/for polling.To show results in reverse order, add the "reverse" argument:http://localhost:8000/?reverse=1If you'd like to tweak or test some functionality, you can modify files in the demo directory and restart the web service:docker compose --project-directory demo restart webAfter the demo, you can revert with:docker compose --project-directory demo down --rmi all -vAlternately, you can run with a venv:python3 -m venv .venv
. .venv/bin/activate
.venv/bin/pip install -r demo/requirements.txt
PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONPATH=.:src DJANGO_SETTINGS_MODULE=demo.settings .venv/bin/celery -A demo.tasks worker --loglevel=INFOTo test SSE, in a new terminal run:PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONPATH=.:src .venv/bin/granian --interface asgi --port 8000 demo.asgi:applicationTo test polling, in a new terminal run:# PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONPATH=.:src DJANGO_SETTINGS_MODULE=demo.settings .venv/bin/django-admin runserver 8001PrerequisitesWe assume you already have these installed and workingA running Django project with Redis configuredA runnning background processor such as celery, with a long-running task definedInstallation:pip install pluto-rtAddpluto_rtto the list of installed apps in project settings.To enable the internal API (required), add this to your top-level urls.py:path("rt_messages/", include("pluto_rt.urls")),UsageThere are two views you'll need to control: the view that kicks off the process
and passes tasks to the background worker, and the view that consumes the results.After following the installation steps above...In thelaunching view:Make the queue nameLaunch the long-running process, passing it the queue namePass the queue name onto the next view (e.g. with a page display or redirect)In thelong-running task:Get the queue handle usingget_rt_queue_handle(queue_name). Put messages items onto the queue. The messages must be pickle-able.When you are finished with the task (successfully or not), callcomplete()on the queue handle.In theresults template:Create a target div with an id, which will hold the formatted DOM elements representing each message.Include the appropriate pluto_rt template (usually pluto_rt/sse.html for ASGI servers, and pluto_rt/polling.html for WSGI). It is recommended that you pass the target directly into the{% include %}tag, since it is often right nearby, e.g.:<ul id="results" class="list-group"></ul>
{% include "pluto_rt/sse.html" with target="#results" %}Create an"item" templatein your template dirs:By default, the system will look forpluto_rt/item.htmlvia your django TEMPLATES setting, however you can pass your own name into your urls path function with theitem_templateargument, such as:path("rt_messages/", include("pluto_rt.urls"), {"item_template": "demo/pluto_rt_item.html"})The format of the template is up to you! The object delivered in the template is named "item". It is a regular django template item at this point, but it will turn into an HTML snippet before it is delivered to the client. It could be as simple as or as complex as you want, so long as it can be delivered as mime type html/text:<div>{{ item }}</div>Note: Instead of writing your own results template & item template, you can
render your view directly to apre-defined templatepluto_rt/table.htmlfor
polling mode only. This is marked as deprecated as it doesn't give you the option to customize
your look and feel, but may be useful for testing.Stop pollingIf you callcomplete()on the queue, the view will return a message that tells htmx to stop polling. So in your processing function, be sure
to call that function when the work is done.Distribution and licenseCreating this as a private ES github repo first, for re-usability,
with intention to secure permission to open source it in the future.When this is pip-installed, it will install the wheel, which means you need to recompile the wheel after making changes.One-time only:pip install build
pip install twineAfter final commit, change the version in pyproject.toml, then runpython3 -m buildThen commit the changes and publish the update to pypi with:twine upload dist/pluto_rt-0.1.2.tar.gz(replacing the actual build number).Versions0.1.0 Initial version0.2.0 Sep 1, 2023: Replaced defunct QR3 lib with our own queueing code (made pluto self-sufficient).
Introduced support for including template partials rather than full-page only.0.3.0 Dec 14, 2023: Added reverse option, complete() function, demo app.0.4.0 Feb 21, 2024: Server-sent events, more flexible templates