Package: pyqs
WARNING: This library is still in beta. It has a stable API and has been deployed in production, but we have not received feedback from a large number of use cases, and it is possible there are unknown bugs.PyQS is a simple task manager for SQS. It’s goal is to provide a simple
and reliablecelery-compatible
interface to working with SQS. It usesboto3under the hood toauthenticateand talk to SQS.InstallationPyQSis available fromPyPIand can
be installed in all the usual ways. To install viaCLI:$pipinstallpyqsOr just add it to yourrequirements.txt.UsagePyQS uses some very simple semantics to create and read tasks. Most of
this comes from SQS having a very simple API.Creating TasksAdding a task to queue is pretty simple.frompyqsimporttask@task(queue='email')defsend_email(subject,message):passsend_email.delay(subject='Hi there')NOTE:This assumes that you have your AWS keys in the appropriate
environment variables, or are using IAM roles. PyQS doesn’t do anything
too special to talk to AWS, it only creates the appropriatebotoconnection.If you don’t pass a queue, PyQS will use the function path as the queue
name. For example the following function lives inemail/tasks.py.@task()defsend_email(subject):passThis would show up in theemail.tasks.send_emailqueue.You can also specify the function path if you want to reference a function in a different project:@task(custom_function_path="foo.bar.send_email")# This references function send_email in foo/bar.py instead of email/tasks.pydefsend_email(subject):passReading TasksTo read tasks we need to run PyQS. If the task is already in yourPYTHON_PATHto be imported, we can just run:$pyqsemail.tasks.send_emailIf we want want to run all tasks with a certain prefix. This is based on
Python’sfnmatch.$pyqsemail.*We can also read from multiple different queues with one call by
delimiting with commas:$pyqssend_email,read_email,write_emailIf you want to run more workers to process tasks, you can up the
concurrency. This will spawn additional processes to work through
messages.$pyqssend_email--concurrency10Simple Process WorkerTo use a simpler version of PyQS that deals with some of the edge cases in the original implementation, pass thesimple-workerflag.$pyqssend_email--simple-workerThe Simple Process Worker differs in the following way from the original implementation.Does not use an internal queue and removes support for theprefetch-multiplierflag. This helps simply the mental model required, as messages are not on both the SQS queue and an internal queue.When thesimple-workerflag is passed, the defaultbatchsizeis 1 instead of 10. This is configurable.Does not check the visibility timeout when reading or processing a message from SQS.Allowing the worker to process the message even past its visibility timeout means we solve the problem of never processing a message ifmax_receives=1and we incorrectly set a shorter visibility timeout and exceed the visibility timeout. Previously, this message would have ended up in the DLQ, if one was configured, and never actually processed.It increases the probability that we process a message more than once, especially ifbatchsize > 1, but this can be solved by the developer checking if the message has already been processed.HooksPyQS has an event registry which can be used to run a function before or after every tasks runs.frompyqsimporttask,eventsdefprint_pre_process(context):print({"pre_process":context})defprint_post_process(context):print({"pre_process":context})events.register_event("pre_process",print_pre_process)events.register_event("post_process",print_post_process)@task(queue="my_queue")defsend_email(subject):passOperational NotesDead Letter QueuesIt is recommended to use aDead Letter Queuefor any queues that are managed by PyQS.  This is because the current strategy
for fetching messages does not delete them upon initial receipt.  A message isONLYdeleted from SQS upon successful completion.This is probably
unexpected behavior if you are coming from Celery with SQS.Celery attempted
to manage this behavior internally, with varying success.If an error arises during message processing, it will be discarded and will
re-appear after the visibility timeout.  This can lead to behavior where
there are messages that will never leave the queue and continuously throw
errors.  A Dead Letter Queue helps resolve this by collecting messages that
have be retried a specified number of times.Worker SeppukuEach process worker will shut itself down after100tasks have been
processed (or failed to process). This is to prevent issues with stale
connections lingering and blocking tasks forever. In addition it helps
guard against memory leaks, though in a rather brutish fashion. After
the process worker shut itself down the managing process should notice
and restart it promptly. The value of100is currently hard-coded,
but could be configurable.Queue BlockingWhile there are multiple workers for reading from different queues, they
all append to the same internal queue. This means that if you have one
queue with lots of fast tasks, and another with a few slow tasks, they
can block eachother and the fast tasks can build up behind the slow
tasks. The simplest solution is to just run two differentPyQScommands, one for each queue with appropriate concurrency settings.Visibility TimeoutCare is taken to not process messages that have exceeded the visibility
timeout of their queue. The goal is to prevent double processing of
tasks. However, it is still quite possible for this to happen since we
do not use transactional semantics around tasks. Therefore, it is
important to properly set the visibility timeout on your queues based on
the expected length of your tasks. If the timeout is too short, tasks
will be processed twice, or very slowly. If it is too long, ephemeral
failures will delay messages and reduce the queue throughput
drastically. This is related to the queue blocking described above as
well. SQS queues are free, so it is good practice to keep the messages
stored in each as homogenous as possible.CompatibilityCelery:PyQS was created to replace celery inside of our infrastructure. To
achieve this goal we wanted to make sure we were compatible with the
basic Celery APIs. To this end, you can easily start trying out PyQS in
your Celery-based system. PyQS can read messages that Celery has written
to SQS. It will readpickleandjsonserialized SQS messages
(Although we recommend JSON).Operating Systems:UNIX. Due to the use of theos.getppidsystem call. This feature can
probably be worked around if anyone actually wants windows support.Boto3:Currently PyQS only supports a few basic connection parameters being
explicitly passed to the connection. Any workboto3does to
transparently find connection credentials, such as IAM roles, will still
work properly.When running PyQS from the command-line you can pass--region,--access-key-id, and--secret-access-keyto override the default
values.CaveatsDurability:When we read a batch of messages from SQS we attempt to add them to our
internal queue until we exceed the visibility timeout of the queue. Once
this is exceeded, we discard the messages and grab a new batch.
Additionally, when a process worker gets a message from the internal
queue, the time the message was fetched from SQS is checked against the
queues visibility timeout and discarded if it exceeds the timeout. The
goal is to reduce double processing. However, this system does not
provide transactions and there are cases where it is possible to process
a message whos’ visibility timeout has been exceeded. It is up to you to
make sure that you can handle this edge case.Task Importing:Currently there is not advanced logic in place to find the location of
modules to import tasks for processing. PyQS will try usingimportlibto get the module, and then find the task inside the
module. Currently we wrap our usage of PyQS inside a Django admin
command, which simplifies task importing. We call the**_main()**method directly, skippingmain()since it only performs argument
parsing.Running inside of containersPyQS assumes that the process id is not 1. If you are running PyQS inside of a
container, you should wrap it in supervisor or something likedummy-init.Why not just use Celery?We like Celery. We(Yipit.com)even
sponsored theoriginal SQS
implementation.
However, SQS is pretty different from the rest of the backends that
Celery supports. Additionally the Celery team does not have the
resources to create a robust SQS implementation in addition to the rest
of their duties. This means the SQS is carrying around a lot extra
features and a complex codebase that makes it hard to debug.We have personally experienced some very vexing resource leaks with
Celery that have been hard to trackdown. For our use case, it has been
simpler to switch to a simple library that we fully understand. As this
library evolves that may change and the the costs of switching may not
be worth it. However, we want to provide the option to others who use
python and SQS to use a simpler setup.Changelog1.0.1Add MessageId for tracking task executions1.0.0Drop Py2 supportAdd new SimpleProcessWorker (https://github.com/spulec/PyQS/pull/76)0.1.6Fix broken pickle of botocore clients.0.1.5Add events hooks for pre and post processors.0.1.4Improve behavior when a queue is created after a worker has started. The worker will now refresh the queues every 30 seconds to check for new queues.0.1.3Change PPID checking to check for actual parent ID, instead ofPID 1.  This fixes issues running on docker containers where PPID of 1 is expected.0.1.2419ce2e Merge pull request #56 from orangain/honor-aws-region7c793d0 Merge pull request #55 from orangain/fix-indentation-error0643fbb Honor aws region configured by .aws/config or env varf5c1db9 Fix indentation errorcdae257 Merge pull request #52 from cedarai/mastera2ac378 Merge pull request #53 from p1c2u/fix/nosetest-remove-stop-parameterdbaa391 Merge pull request #51 from p1c2u/fix/pep8-styles-fixes1577382 Nosetest remove stop parameterb7420e3 Add current directory to PYTHONPATH8d04b62 Graceful shutdown logging msg fix796acbc PEP8 styles fixes72dcb62 Merge pull request #50 from hobbsh/add_exampled00d31f Update readmedfbf459 Use .delay() to submit messages612158f Merge pull request #49 from hobbsh/no_log_0_msg09a649f Use logger.debug for success SQS log linedfd56c3 Fix typos in readmea774155 Add example flask app17e7b7c Don’t log message retrieve success when there are 0 messages14eb827 Add shutdown signal logging.0.1.1Fix KeyError on accounts without queues0.1.0Upgrade to boto30.0.22Fix Python 3 supportAllow overwriting thedelay_secondsattibute at call time0.0.21Add ability to tunePREFETCH_MULTIPLIERwith--prefetch-multiplier.0.0.20Respect--batch-sizewhen sizing internal queue on ManagerWorker0.0.19Add ability to run with tunable BATCHSIZE and INTERVAL. Thanks to @masayangAdd ability to create tasks with a visibility delay.  Thanks to @joshbuddyAdd ability to create tasks with a custom function location, allowing cross project tasks0.0.18Convert Changelog to .rstAdd Changelog to long description on Pypi.  Thanks to @adamchainz0.0.17Fix typos in READMEAdd notes on Dead Letter Queues to README0.0.16Switch README to reStructuredText (.rst) format so it renders on PyPI0.0.15Process workers will kill themselves after attempting to process 100
requests, instead of checking the internal queue 100 times.If we find no messages on the internal queue, sleep for a moment
before rechecking.0.0.14Process workers will kill themselves after processing 100 requestsProcess workers will check a message’s fetch time and visibility
timeout before processing, discarding it if it has exceeded the
timeout.Log theprocess_time()used to process a task to the INFO level.0.0.13Only pass SQS Queue ID to internal queue. This is attempting to fix a
bug when processing messages from multiple queues.0.0.12Remove extraneous debugging code0.0.11Add additional debugging to investigate message deletion errors0.0.10Give each process worker its own boto connection to avoid
multiprocess race conditions during message deletion0.0.9Change long polling interval to a valid value, 0<=LPI<=200.0.8Switched to long polling when pulling down messages from SQS.Moved message deletion from SQS until after message has been
processed.0.0.7Added capability to read JSON encoded celery messages.0.0.6Switched shutdown logging to INFOAdded brief sleep to message retrieval loop so that we don’t look
like we are using a ton of CPU spinning.0.0.5Switching task failure logging to ERROR (actually this time)Moved task success logging to INFOAdded INFO level logging for number of messages retrieved from an SQS
queue.Moved Reader and Worker process counts to DEBUG0.0.4Added ability to passregion,access_key_idandsecret_access_keythrough to Boto when creating connectionsSwitched logging of task failure to theERRORlogger, fromINFO.