Package: django-outbox-pattern
Django outbox patternA django application to make it easier to use
thetransactional outbox patternInstallationInstall django-outbox-pattern with pippipinstalldjango-outbox-patternAdd to settings# settings.pyINSTALLED_APPS=["django_outbox_pattern",]DJANGO_OUTBOX_PATTERN={"DEFAULT_STOMP_HOST_AND_PORTS":[("127.0.0.1",61613)],"DEFAULT_STOMP_USERNAME":"guest","DEFAULT_STOMP_PASSCODE":"guest",}Rum migrationspythonmanage.pymigrateUsage/ExamplesThepublishdecorator adds theoutbox tableto the model.publishaccepts list of Config. The Config accepts four params thedestinationwhich is required,fieldswhich the default are all the fields of the model,serializerwhich by default adds theidin the message
to be sent andversionwhich by default is empty.Note:fieldsandserializerare mutually exclusive, serializer overwrites the fields.The Config typingfromtypingimportListfromtypingimportNamedTuplefromtypingimportOptionalclassConfig(NamedTuple):destination:strfields:Optional[List[str]]=Noneserializer:Optional[str]=Noneversion:Optional[str]=NoneOnly destination in configfromdjango.dbimportmodelsfromdjango_outbox_pattern.decoratorsimportConfigfromdjango_outbox_pattern.decoratorsimportpublish@publish([Config(destination='/topic/my_route_key')])classMyModel(models.Model):field_one=models.CharField(max_length=100)field_two=models.CharField(max_length=100)This generates the following data to be sent.Published(destination='/topic/my_route_key', body='{"id": 1, "field_one": "Field One", "field_two": "Field Two"}')Change destination version in configfromdjango.dbimportmodelsfromdjango_outbox_pattern.decoratorsimportConfigfromdjango_outbox_pattern.decoratorsimportpublish@publish([Config(destination='/topic/my_route_key',version="v1")])classMyModel(models.Model):field_one=models.CharField(max_length=100)field_two=models.CharField(max_length=100)This generates the following data to be sent.Published(destination='/topic/my_route_key.v1', body='{"id": 1, "field_one": "One", "field_two": "Two"}', version="v1")With destinations and fieldsfromdjango.dbimportmodelsfromdjango_outbox_pattern.decoratorsimportConfigfromdjango_outbox_pattern.decoratorsimportpublish@publish([Config(destination='/topic/my_route_key',fields=["field_one"])])classMyModel(models.Model):field_one=models.CharField(max_length=100)field_two=models.CharField(max_length=100)This generates the following data to be sent.Published(destination='/topic/my_route_key', body='{"id": 1, "field_one": "Field One"}')With destinations and serializerfromdjango.dbimportmodelsfromdjango_outbox_pattern.decoratorsimportConfigfromdjango_outbox_pattern.decoratorsimportpublish@publish([Config(destination='/topic/my_route_key',serializer='my_serializer')])classMyModel(models.Model):field_one=models.CharField(max_length=100)field_two=models.CharField(max_length=100)defmy_serializer(self):return{"id":self.id,"one":self.field_one,"two":self.field_two}This generates the following data to be sent.Published(destination='/topic/my_route_key', body='{"id": 1, "one": "Field One", "two": "Field Two"}')With multi destinations and serializersfromdjango.dbimportmodelsfromdjango_outbox_pattern.decoratorsimportConfigfromdjango_outbox_pattern.decoratorsimportpublish@publish([Config(destination='/topic/my_route_key_1',serializer="my_serializer_1"),Config(destination='/topic/my_route_key_2',serializer="my_serializer_2"),])classMyModel(models.Model):field_one=models.CharField(max_length=100)field_two=models.CharField(max_length=100)defmy_serializer_1(self):return{"id":self.id,"one":self.field_one,}defmy_serializer_2(self):return{"id":self.id,"two":self.field_two}This generates the following data to be sent.Published(destination='/topic/my_route_key_1', body='{"id": 1, "one": "Field One"}')
Published(destination='/topic/my_route_key_2', body='{"id": 1, "two": "Field Two"}')Publish/Subscribe commandsPublish commandTo send the messages added to the Published model it is necessary to start the producer with the following command.pythonmanage.pypublishPublish message via outboxIt is possible to use the outbox pattern with a custom logic before sending the message to the outbox table.fromdjango.dbimporttransactionfromdjango_outbox_pattern.modelsimportPublisheddefcustom_business_logic()->None:# run your custom business logicwithtransaction.atomic():YourBusinessModel.objects.create()Published.objects.create(destination="your_destination",body={"some":"data"})With this you can ensure that the messages can be published in the same database transaction of your business logic.Publish message directlyIt is possible to send messages directly without using the outbox table# send.pyfromdjango_outbox_pattern.factoriesimportfactory_producerdefsend_event(destination,body,headers):withfactory_producer()asproducer:producer.send_event(destination=destination,body=body,headers=headers)Subscribe commandConsumers created through the library implement the idempotency pattern using the header attributemessage-id. The library configures it as unique in the database. This ensures a given message is only processed once, no matter what.
To correctly implement this, you must open a transaction with the database to persist the data from your logic and execute thesavemethod of thepayloadobject. Once the code is performed correctly, the library guarantees the message is removed from the broker.If you need to discard the message due to a business rule, use thenackmethod of the Payload object. This call removes the message from the broker. This method performs no persistence in the database and can be called outside your database transaction. If it fails for any reason, the message is resent to the consumer.Alert:You need to use eithersaveornackto process of your message. The library cannot make the decision for the developer, and it is up to the developer to determine whether to use thesaveornackmethod. However, in case of an exception during the operation, thenackmethod will be triggered automaticallyThe same service (code + database) cannot consume the same message even with different consumers.Create a function that receives an instance ofdjango_outbox_pattern.payloads.Payload# callbacks.pyfromdjango.dbimporttransactionfromdjango_outbox_pattern.payloadsimportPayloaddefcallback(payload:Payload):ifmessage_is_invalid(payload.body):payload.nack()returnwithtransaction.atomic():persist_your_data()payload.save()To start the consumer, after creating the callback, it is necessary to execute the following command.pythonmanage.pysubscribe'dotted.path.to.callback` 'destination' 'queue_name'The command takes three parameters:callback: the path to the callback function.destination: the destination where messages will be consumed following one of thestomppatternsqueue_name(optional): the name of the queue that will be consumed. If not provided, the routing_key of the destination will be used.SettingsDEFAULT_CONNECTION_CLASSThe stomp.py class responsible for connecting to the broker. Default:stomp.StompConnection12DEFAULT_CONSUMER_LISTENER_CLASSThe consumer listener class. Default:django_outbox_pattern.listeners.ConsumerListenerDEFAULT_GENERATE_HEADERSA function to add headers to the message. Default:django_outbox_pattern.headers.generate_headersDEFAULT_MAXIMUM_BACKOFF:Maximum wait time for connection attempts in seconds. Default:3600(1 hour)DEFAULT_MAXIMUM_RETRY_ATTEMPTSMaximum number of message resend attempts. Default:50DEFAULT_PAUSE_FOR_RETRYPausing for attempts to resend messages in seconds. Defualt:240(4 minutes)DEFAULT_WAIT_RETRYTime between attempts to send messages after the pause. Default:60(1 minute)DEFAULT_PRODUCER_LISTENER_CLASS:The producer listener class. Default:django_outbox_pattern.listeners.ProducerListenerDEFAULT_STOMP_HOST_AND_PORTSList of host and port tuples to try to connect to the broker. Default[("127.0.0.1", 61613)]DEFAULT_STOMP_QUEUE_HEADERSHeaders for queues. Default:{"durable": "true", "auto-delete": "false", "prefetch-count": "1"}DEFAULT_STOMP_HEARTBEATSTime tuples for input and output heartbeats. Default:(10000, 10000)DEFAULT_STOMP_VHOSTVirtual host. Default: "/"DEFAULT_STOMP_USERNAMEUsername for connection. Default:"guest"DEFAULT_STOMP_PASSCODEPassword for connection. Default:"guest"DEFAULT_STOMP_USE_SSLFor ssl connections. Default: FalseDEFAULT_STOMP_KEY_FILEThe path to a X509 key file. Default: NoneDEFAULT_STOMP_CERT_FILEThe path to a X509 certificate. Default: NoneDEFAULT_STOMP_CA_CERTSThe path to the a file containing CA certificates to validate the server against.
If this is not set, server side certificate validation is not done. Default: NoneDEFAULT_STOMP_CERT_VALIDATORFunction which performs extra validation on the client certificate, for example
checking the returned certificate has a commonName attribute equal to the
hostname (to avoid man in the middle attacks).
The signature is: (OK, err_msg) = validation_function(cert, hostname)
where OK is a boolean, and cert is a certificate structure
as returned by ssl.SSLSocket.getpeercert(). Default: NoneDEFAULT_STOMP_SSL_VERSIONSSL protocol to use for the connection. This should be one of the PROTOCOL_x
constants provided by the ssl module. The default is ssl.PROTOCOL_TLSv1.DEFAULT_STOMP_SSL_PASSWORDSSL passwordDEFAULT_EXCLUSIVE_QUEUEFor exclusive queue feature. Default: FalseDAYS_TO_KEEP_DATAThe total number of days that the system will keep a message in the database history. Default: 30REMOVE_DATA_CACHE_TTLThis variable defines the time-to-live (TTL) value in seconds for the cache used by the_remove_old_messagesmethod in thedjango_outbox_patternapplication. The cache is used to prevent the method from deleting old data every time it is run, and the TTL value determines how long the cache entry should remain valid before being automatically deleted. It can be customized by setting the REMOVE_DATA_CACHE_TTL variable. Default: 86400 seconds (1 day)OUTBOX_PATTERN_PUBLISHER_CACHE_KEYTheOUTBOX_PATTERN_PUBLISHER_CACHE_KEYvariable controls the key name of the cache used to store the outbox pattern publisher. Default:remove_old_messages_django_outbox_pattern_publisher.OUTBOX_PATTERN_CONSUMER_CACHE_KEYTheOUTBOX_PATTERN_CONSUMER_CACHE_KEYvariable controls the key name of the cache used to store the outbox pattern publisher. Default:remove_old_messages_django_outbox_pattern_consumer.