Package: valer.queue
Queue of asynchronous tasks for SENAITE LIMSAboutThis add-on enables asynchronous tasks forSENAITE LIMS, that allows to
better handle concurrent actions and processes when the workload is high. Is
specially indicated for high-demand instances and for when there are custom
processes that take long to complete. Essentially,senaite.queuereduces the
chance of transaction commits by handling tasks asynchronously, in an
unattended and sequential manner.Once installed, this add-on enables asynchronous processing of those tasks that
usually have a heavier footprint regarding performance, and with highest chance
of transaction conflicts:Assignment of analyses to worksheetsAssignment of worksheet template to a worksheetCreation of a worksheet by using a worksheet templateWorkflow actions (submit, verify, etc.) for analyses assigned to worksheetsRecursive permissions assignment on client contacts creationThis add-on neither provides support for workflow transitions/actions at Sample
level nor for Sample creation. However, this add-on can be extended easily to
match additional requirements.Documentationhttps://senaitequeue.readthedocs.ioContributeWe want contributing to SENAITE.QUEUE to be fun, enjoyable, and educational
for anyone, and everyone. This project adheres to theContributor Covenant.By participating, you are expected to uphold this code. Please report
unacceptable behavior.Contributions go far beyond pull requests and commits. Although we love giving
you the opportunity to put your stamp on SENAITE.QUEUE, we also are thrilled
to receive a variety of other contributions.Please, readContributing to senaite.queue document.If you wish to contribute with translations, check the project site onTransifex.Feedback and supportCommunity siteGitter channelUsers listLicenseSENAITE.QUEUECopyright (C) 2019-2020 RIDING BYTES & NARALABSThis program is free software; you can redistribute it and/or modify it under
the terms of theGNU General Public License version 2as published
by the Free Software Foundation.This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.Release notesUpdate from 1.0.1 to 1.0.2With version 1.0.2, the legacy storage for queued tasks has changed and helper
storages (e.g. for Worksheets) are no longer required.IQueuedmarker
interface is no longer used neither. Most of the base code has been refactored
keeping in mind the following objectives:Less complexity: less code, better codeLess chance of transaction commit conflictsBoost performance: better experience, with no delaysAll these changes also makes the add-on easier to extend and maintain. The
downside is that old legacy storage is no longer used and therefore, tasks that
were queued before the upgrade will be discarded.Be sure there are no remaining tasks in the queue before the upgradeIf you have your own add-on extendingsenaite.queue, please review the changes
and check if some parts of your add-on require modificationsA queue server has been introduced. Therefore, two zeo clients are recommended:
one that acts as the server and at least another one in charge of consuming
tasks. Also, this version now depends on three additional packages:requests,senaite.jsonapiandcryptography. Please read the installation
instructions and run buildout to download the dependencies.InstallationIs strongly recommended to have a SENAITE instance setup in ZEO mode, because
this add-on is especially useful when a reserved zeo client is used to act as
a queue server and at least one additional zeo client for tasks consumption.In standalone installation, only one CPU / CPU core can be used for processing
requests, with a limited number of threads (usually 2). With a ZEO mode setup,
the database can be used by multiple zeo clients at the same time, each one
using it’s own CPU. SeeScalability and ZEOfor further information.Create a new reserved user in SENAITE instance (under/senaite/acl_users). The
recommended username isqueue_consumer.This user will be used by the consumer to pop tasks from the queue server in a
sequential manner. The consumer will eventually process the task, but acting as
the user who initially triggered the process. However, the reserved user
responsible of dispatching must have enough privileges. Assign this user to
the group “Site Administrator” and/or “Manager”.First, add this add-on in theeggssection of your buildout configuration file:[buildout]...[instance]...eggs=...senaite.queueThen, add a two clients (a consumer and the server) in your buildout
configuration:# Reserved user for dispatching queued tasks# See https://pypi.org/project/senaite.queuequeue-user-name=queue_consumerqueue-user-password=queue_consumer_passwordparts=....queue_consumerqueue_serverand configure two reserved clients:[queue_consumer]# ZEO Client reserved for the consumption of queued tasks<=client_baserecipe=plone.recipe.zope2instancehttp-address=127.0.0.1:8089zope-conf-additional=<clock-server>method /senaite/queue_consumeperiod 5user ${buildout:queue-user-name}password ${buildout:queue-user-password}host localhost:8089</clock-server>[queue_server]# ZEO Client reserved to act as the server of the queue<=client_baserecipe=plone.recipe.zope2instancehttp-address=127.0.0.1:8090NoteThese clients will listen to ports 8089 and 8090. They should not be
accessible to regular users. Thus, if you use a load-balancer
(e.g HAProxy), is strongly recommended to not add these clients in
the backend pool.In most scenarios, this configuration is enough. However, senaite.queue supports
multi consumers, that can be quite useful for those SENAITE installations that
have a very high overload. To add more consumers, add as many zeo client
sections as you need with the additionalclock-serverzope configuration. Do
not forget to set the valuehostcorrectly to all them, because this value is
used by the queue server to identify the consumers when tasks are requested.The maximum number of concurrent consumers supported by the queue server is 4.Runbin/buildoutafterwards. With this configuration, buildout will download
and install the latest published release ofsenaite.queue from Pypi.NoteIf the buildout fails with aImportError: cannot import name aead,
please update OpenSSL to v1.1.1 or above. OpenSSL v1.0.2 is no longer
supported bycryptographystarting from v3.2. Please, read thechangelog from cryptographyfor further information. Although not
recommended, you can alternatively stick to version 3.1.1 by addingcryptography=3.1.1in[versions]section from your buildout.Once buildout finishes, start the clients:$sudo-uplone_daemonbin/client1start$sudo-uplone_daemonbin/queue_serverstart$sudo-uplone_daemonbin/queue_clientstartNoteplone_daemonis the default user created by the quick-installer
when installing Plone in ZEO cluster mode. Please checkInstallation of Plonefor further information. You might need to
change this user name depending on how you installed SENAITE.Then visit your SENAITE site and login with a user with “Site Administrator”
privileges to activate the add-on:http://localhost:8080/senaite/prefs_install_products_formNoteIt assumes you have a SENAITE zeo client listening to port 8080Once activated, go toSite Setup > Queue Settingsand, in field “Queue Server”,
type the url of the zeo client that will act as the server of the queue.http://localhost:8090/senaiteNoteDo not forget to specify the site id in the url (usually “senaite”)Changelog1.0.3 (unreleased)Fix client’s queue tasks in “queued” status are not updated when “running”1.0.2 (2020-11-15)Support for multiple consumers (up to 4 concurrent processes)Added JSON API endpoints for both queue server and clientsQueue server-client implementation, without the need of annotationsAdded PAS plugin for authentication, with symmetric encryptionDelegate the reindex object security to queue when linking contacts to users#7 Allow to queue generic worflow actions without specific adapter#7 Redux and better performance#6 Allow the prioritization of tasks#5 No actions can be done to worksheets with queued jobs1.0.1 (2020-02-09)Allow to manually assign the username to the task to be queuedSupport for failed tasksNotify when the value for max_seconds_unlock is too low#3 Newqueue_tasksview with the list of tasks and statistics#2 Add max_retries setting for failing tasks#1 Add sample guard to prevent transitions when queued analyses1.0.0 (2019-11-10)First version