Package: aws-sqs-ext-client
aws-sqs-ext-clientThe Amazon SQS Extended Client Library for Python for sending and receiving large messages via S3. This aims to have the same capability ofAmazon SQS Extended Client Library for Java, in which the client can send and receive messages larger than the SQS limit (256 KB), up to the limit of S3 (5 TB), in the similar way toBoto3 - The AWS SDK for Python. This library supports:Send/receive large messages over than threshold (by default, it's 2**18)Enable to send/receive all messages, even though the data size is under the threshold, by turning onalways_through_s3Enable to configure the threshold to which size you wantEnable to check message's MD5 chechsum when receiving the large messageEnalbe to configure the S3 bucket, like its ACL, where the large messages are temporarily storedPrerequisitesThis package requires AWS account and Python 3.7+ environment. Please configure an AWS account as well as prepare the Python by referring README ofboto3. Or, just an example,aws-vaultis the useful tool to handle AWS account, likeaws-vault exec PROFILE_USER -- python APP_WITH_THIS_LIB.Installationpipinstallaws-sqs-ext-clientUsageThis section shows some of examples to use this library. Please seetest/integration/test_all.pyto know more.Extended methodsThe table below shows extended methods to send/receive/delete large messages. Those APIs have same specifications as methods without "_extended" described inSQS - boto3 documentation. For instance,send_message_extendedof the client API accepts the same arguments assend_messageof the client API.TypesMethodsDescriptionClientsend_message_extendedsend one large messageClientreceive_message_extendedreceive multiple large messages (with MaxNumberOfMessages)Clientdelete_message_extendeddelete one large messageClientsend_message_batch_extendedsend multiple large messagesClientdelete_message_batch_extendeddelete multiple large messagesResource (Queue)send_message_extendedsend one large messageResource (Queue)receive_messages_extendedreceive multiple large messages (with MaxNumberOfMessages)Resource (Message)delete_extendeddelete one large messageResource (Queue)send_messages_extendedsend multiple large messagesResource (Queue)delete_messages_extendeddelete multiple large messagesSession InitializationFirst of all, you need to initialize and extend the boto3 session.importboto3# override boto3.session.Session and overwrite the default sessionimportaws_sqs_ext_client# noqa: F401# create session# instead, you can use boto3.DEFAULT_SESSIONsession=boto3.session.Session()# extend the session# can add the following options# always_through_s3: bool: enable to store even small message into S3 (by default, it's False)# message_size_threshold: int: like 2*10. enable to change the threshold (default value is 2**18)# s3_bucket_params: dict: add parameters to create/check the bucket where this lib stores the messages.#   By default, this parameter is `{'ACL': 'private'}`.#   If you already created S3 bucket for storing huge messages and utilize it, set `s3_bucket_params=None`.#   With non-None parameter, if you don't specify AWS_DEFAULT_REGION on the environment variables,#   you need to specify the location constrain by#   {'CreateBucketConfiguration': {'LocationConstraint': YOUR_REGION}}.#   Available other parameters are shown in#   https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.create_bucket## It's recommended to create a bucket for object storing preliminarily even though this module gives you automatic creation functionality.# That's because you should create a bucket with some options, like the specific finite object lifecycle configured by `put_bucket_lifecycle_configuration`.session.extend_sqs('S3_BUCKET_NAME_TO_STORE_MESSAGES')with Resource# please initialize session like abovemessage='large string message more than threshold'# create/get queue# you can create both standard and fifo queuesqs=session.resource('sqs')queue=sqs.create_queue(QueueName='test',Attributes={'DelaySeconds':'5'})# or using existing queuequeue=sqs.get_queue_by_name(QueueName='test')# send message# you can add any other arguments that are accepted in `send_message`,# like `MessageAttributes` and `MessageDeduplicationId`res=queue.send_message_extended(MessageBody=message)# receive message# you can add any other arguments that are accepted in `receive_messages`,# like `VisibilityTimeout`received=queue.receive_messages_extended(MessageAttributeNames=['All'],MaxNumberOfMessages=10,WaitTimeSeconds=5)forrinreceived:# if you want, you can check received message with given MD5# the function `checkdata` should be given by youcheckdata(r.body,r.meta.data['MD5OfBody'])checkdata(r.meta.data['MessageAttributes'],r.meta.data['MD5OfMessageAttributes'])# process whatever you want with a message# delete both a message from the queue and a data on S3 bucket# this should be called til visibility timeout is elapsed# see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.htmlr.delete_extended()with Client# please initialize session like abovemessage='large string message more than threshold'# create/get queue# you can create both standard and fifo queuesqs=session.client('sqs')queue=sqs.create_queue(QueueName='test',Attributes={'DelaySeconds':'5'})# or using existing queuequeue=sqs.get_queue_by_name(QueueName='test')# send message# you can add any other arguments that are accepted in `send_message`,# like `MessageAttributes` and `MessageDeduplicationId`res=sqs.send_message_extended(QueueUrl=queue['QueueUrl'],MessageBody=message)# receive message# you can add any other arguments that are accepted in `receive_messages`,# like `VisibilityTimeout`received=sqs.receive_message_extended(QueueUrl=queue['QueueUrl'],MessageAttributeNames=['All'],MaxNumberOfMessages=10,WaitTimeSeconds=5)received=received['Messages']if'Messages'inreceivedelse[]forrinreceived:# if you want, you can check received message with given MD5# the function `checkdata` should be given by youcheckdata(r['Body'],r['MD5OfBody'])checkdata(r['MessageAttributes'],r['MD5OfMessageAttributes'])# process whatever you want with a message# delete both a message from the queue and a data on S3 bucket# this should be called til visibility timeout is elapsed# see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.htmlsqs.delete_message_extended(QueueUrl=queue['QueueUrl'],ReceiptHandle=r['ReceiptHandle'])with Resource (multiple sending/deleting)With multiple deletion methoddelete_messages_extended, please use receipt handles gotten fromsqs.Message.meta.data['ReceiptHandle']instead ofsqs.Message.receipt_handle. Becausesqs.Message.receipt_handleis read-only attributes, the methoddelete_messages_extendedcannot overwrite the "correct" handle. With the right handle fromsqs.Message.meta.data['ReceiptHandle'], the methoddelete_messages_extendeddeletes both messages in the queue and data objects in the S3 bucket. Otherwise, it only deletes messages in the queue.# please initialize session like abovemessages=[{'Id':'1','MessageBody':"large string message more than threshold",},{'Id':'2','MessageBody':"large string message more than threshold",}]# create/get queue# you can create both standard and fifo queuesqs=session.resource('sqs')queue=sqs.create_queue(QueueName='test',Attributes={'DelaySeconds':'5'})# or using existing queuequeue=sqs.get_queue_by_name(QueueName='test')# send messages# you can add any other arguments that are accepted in `send_messages`,# like `MessageAttributes` and `MessageDeduplicationId`res=queue.send_messages_extended(Entries=messages)# receive messages# you can add any other arguments that are accepted in `receive_messages`,# like `VisibilityTimeout`received=queue.receive_messages_extended(MessageAttributeNames=['All'],MaxNumberOfMessages=10,WaitTimeSeconds=5)receipt_handles=[]forrinreceived:# if you want, you can check received message with given MD5# the function `checkdata` should be given by youcheckdata(r.body,r.meta.data['MD5OfBody'])checkdata(r.meta.data['MessageAttributes'],r.meta.data['MD5OfMessageAttributes'])# process whatever you want with a message# aggreage receipt handle: use meta one instad of its attributereceipt_handles.append({'Id':str(i),'ReceiptHandle':r.meta.data['ReceiptHandle']})# delete both messages from the queue and data on S3 bucket# this should be called til visibility timeout is elapsed# see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.htmlres=queue.delete_messages_extended(Entries=receipt_handles)Testtests/integration/test_all.pygives you clues about how to use this module with AWS resources.# AWS credentials, like AWS_ACCESS_KEY_ID, should be set preliminarilypythontests/integration/test_all.pytests/unitsincludes unit tests.exportTEST_THRESHOLD=90pythonsetup.pytestcoveragereport--fail-under=${TEST_THRESHOLD}Lintflake8aws_sqs_ext_client--count--show-source--statisticsLicenseMIT License.