Package: aws-sqs-batchlib
aws-sqs-batchlib for PythonConsume and process Amazon SQS queues in large batches.FeaturesConsume arbitrary number of messages from an Amazon SQS queue.Define maximum batch size and batching window in seconds to receive a batch
of messages from Amazon SQS queue similar to Lambda Event Source Mapping.Send arbitrary number of messages to an Amazon SQS queue.Delete arbitrary number of messages from an Amazon SQS queue.InstallationInstall from PyPI with pippip install aws-sqs-batchlibor with the package manager of choice.Usageaws-sqs-batchlibprovides the following methods:delete_message_batch()- Delete arbitrary number of messages from an Amazon SQS queue.receive_message()- Receive arbitrary number of messages from an Amazon SQS queue.send_message_batch()- Send arbitrary number of messages to an Amazon SQS queue.These methods invoke the corresponding boto3SQS.Clientmethods multiple times to send, receive or delete an arbitrary number of messages from an Amazon SQS queue. They accept the same arguments and have
the same response structure as their boto3 counterparts. See boto3 documentation for more details:delete_message_batch()receive_message()send_message_batch()Receiveimportaws_sqs_batchlib# Receive up-to 100 messages from the given queue, polling the queue for# up-to 15 seconds to fill the batch.res=aws_sqs_batchlib.receive_message(QueueUrl="https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue",MaxNumberOfMessages=100,WaitTimeSeconds=15,)# Returns messages in the same format as boto3 / botocore SQS Client# receive_message() method.assertres=={'Messages':[{'MessageId':'[.]','ReceiptHandle':'AQ[.]JA==','MD5OfBody':'[.]','Body':'[.]'},{'MessageId':'[.]','ReceiptHandle':'AQ[.]wA==','MD5OfBody':'[.]','Body':'[.]'}# ... up-to 100 messages]}Sendimportaws_sqs_batchlib# Send an arbitrary number of messages to a queueres=aws_sqs_batchlib.send_message_batch(QueueUrl="https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue",Entries=[{"Id":"1","MessageBody":"<...>"},{"Id":"2","MessageBody":"<...>","DelaySeconds":1000000},# ...{"Id":"175","MessageBody":"<...>"},# ...],)# Returns result in the same format as boto3 / botocore SQS Client# send_message_batch() method.assertres=={"Successful":[{"Id":"1","MessageId":"<...>","MD5OfMessageBody":"<...>"},# ...],"Failed":[{"Id":"2","SenderFault":True,"Code":"InvalidParameterValue","Message":"Value 1000000 for parameter DelaySeconds is invalid. Reason: DelaySeconds must be >= 0 and <= 900.",}],}Deleteimportaws_sqs_batchlib# Delete an arbitrary number of messages from a queueres=aws_sqs_batchlib.delete_message_batch(QueueUrl="https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue",Entries=[{"Id":"1","ReceiptHandle":"<...>"},{"Id":"2","ReceiptHandle":"<...>"},# ...{"Id":"175","ReceiptHandle":"<...>"},# ...],)# Returns result in the same format as boto3 / botocore SQS Client# delete_message_batch() method.assertres=={"Successful":[{"Id":"1"},# ...],"Failed":[{"Id":"2","SenderFault":True,"Code":"ReceiptHandleIsInvalid","Message":"The input receipt handle is invalid.",}],}DevelopmentRequires Python 3 and Poetry. Useful commands:# Setup environmentpoetryinstall# Run tests (integration test requires rights to create, delete and use DynamoDB tables)maketest# Run lintersmake-klint# Format codemakeformatBenchmarks & Manual TestingUsebenchmark/end2end.pyto benchmark and test the library functionality and performance. Execute following commands in Poetry virtualenv (executepoetry shellto get there):# SetupexportPYTHONPATH=$(pwd)exportAWS_DEFAULT_REGION=eu-north-1# Send, receive and delete 512 messages, run test 5 timespython3benchmark/end2end.py\--queue-urlhttps://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue--num-messages512--iterations5Benchmarks against an Amazon SQS queue on the same AWS region (eu-north-1, c5.large instance) show following
throughput:Send - ~500 to ~800 messages / secondReceive - ~800 to ~1400 messages / secondDelete - ~900 to ~1600 messages / secondLicenseMIT.