Package: barterdude
BarterDudeMessage exchange engine to build pipelines using brokers like RabbitMQ. This project is build on top of the greatasync-worker.InstallUsing Python 3.6+pipinstallbarterdudeUsageBuild your consumer with this complete example:importloggingfrombarterdudeimportBarterDudefrombarterdude.monitorimportMonitorfrombarterdude.hooks.healthcheckimportHealthcheckfrombarterdude.hooks.loggingimportLoggingfrombarterdude.hooks.metrics.prometheusimportPrometheusfrombarterdude.messageimportMessagefrombarterdude.confimportgetLogger# from my_project import MyHook # (you can build your own hooks)# configure RabbitMQbarterdude=BarterDude(hostname="localhost",username="guest",password="guest",prefetch=256,)# Prometheus labels for automatic metricslabels=dict(app_name="my_app",team_name="my_team")healthcheck=Healthcheck(barterdude)# automatic and customizable healthcheckprometheus=Prometheus(labels)# automatic and customizable Prometheus integrationself.logger=getLogger("my_app",logging.DEBUG)# automatic json log with barterdude namespace# BARTERDUDE_DEFAULT_APP_NAME is an env var to control your project namespace# BARTERDUDE_DEFAULT_LOG_LEVEL is an env var to control loglevel by number 0, 10, 20, etc...monitor=Monitor(healthcheck,prometheus,# MyHook(barterdude, "/path"), # (will make localhost:8080/path url)Logging()# automatic and customizable logging)my_metric=prometheus.metrics.counter(name="fail",description="fail again")# It's the same as https://github.com/prometheus/client_python@barterdude.consume_amqp(["queue1","queue2"],monitor,coroutines=10,# number of coroutines spawned to consume messages (1 per message)bulk_flush_interval=60.0,#  max waiting time for messages to start process n_coroutinesrequeue_on_fail=True# should retry or not the message)asyncdefyour_consumer(msg:Message):# you receive only one message and we parallelize processing for youawaitbarterdude.publish_amqp(exchange="my_exchange",data=msg.body)ifmsg.body=="fail":my_metric.inc()# you can use prometheus metricshealthcheck.force_fail()# you can use your hooks inside consumer toomsg.reject(requeue=False)# You can force to reject a message, exactly equal https://b2wdigital.github.io/async-worker/src/asyncworker/asyncworker.rabbitmq.html#asyncworker.rabbitmq.message.RabbitMQMessageifmsg.body=="exception":raiseException()# this will reject the message and requeue# if everything is fine, than message automatically is acceptedbarterdude.run()# you will start consume and start a server on http://localhost:8080# Change host and port with ASYNCWORKER_HTTP_HOST and ASYNCWORKER_HTTP_PORT env varsBuild your own HookBase Hook (Simple One)These hooks are called when message retreive, have success and fail.frombarterdude.hooksimportBaseHookfromasyncworker.rabbitmq.messageimportRabbitMQMessageclassMyCounterHook(BaseHook):_consume=_fail=_success=0asyncdefon_success(self,message:RabbitMQMessage):self._success+=1asyncdefon_fail(self,message:RabbitMQMessage,error:Exception):self._fail+=1asyncdefbefore_consume(self,message:RabbitMQMessage):self._consume+=1Http Hook (Open Route)These hooks can do everything simple hook does, but responding to a route.fromaiohttpimportwebfrombarterdude.hooksimportHttpHookfromasyncworker.rabbitmq.messageimportRabbitMQMessageclassMyCounterHttpHook(HttpHook):_consume=_fail=_success=0asyncdef__call__(self,req:web.Request):returnweb.json_response(dict(consumed=self._consume,success=self._success,fail=self._fail))asyncdefon_success(self,message:RabbitMQMessage):self._success+=1asyncdefon_fail(self,message:RabbitMQMessage,error:Exception):self._fail+=1asyncdefbefore_consume(self,message:RabbitMQMessage):self._consume+=1Data SharingFollowing the approach found inasync-workerand inaiohttp,BarterDudediscourages the use of global variables, aka singletons.To share data states globally in an application,BarterDudebehaves like adict.
As an example, one can save a global-like variable in aBarterDudeinstance:frombarterdudeimportBarterDudebarterdude=BarterDude()baterdude["my_variable"]=dataand get it back in a consumerasyncdefconsumer_access_storage(msg):data=baterdude["my_variable"]Monitoring extra modules on HealthcheckIf you run extra modules on your application, like workers or services, you can include them in the healthcheck.First, you need to update your module to implement the interfaceHealthcheckMonitored:frombarterdude.hooks.healthcheckimportHealthcheckMonitoredclassExtraService(HealthcheckMonitored):Implementing that interface will require the definition of the methodhealthcheckin your module. It should return a boolean value indicating if your module is healhty or not:defhealthcheck(self):returnself._thread.is_alive()Finally, you need to make the BarterDude and Healthcheck module be aware of your module. To do so, you'll use the Data Sharing feature:frombarterdudeimportBarterDudefromapp.extra_serviceimportExtraServicebarterdude=BarterDude()barterdude["extra_service"]=ExtraService()If you are already running your extra modules on BartedDude startup using the data sharing model, it's all done:@app.run_on_startupasyncdefstartup(app):app["client_session"]=ClientSession()app["extra_service"]=ExtraService()The healthcheck module will identify all shared modules that implement the interfaceHealthcheckMonitoredand run its healthcheck method automatically.
The result of all monitored modules will be included in the result body of the healthcheck endpoint and if any of the modules fail, the healthcheck endpoint will indicate that:{"extra_service":"ok","message":"Success rate: 1.0 (expected: 0.9)","fail":0,"success":1,"status":"ok"}Schema ValidationConsumed messages can be validated by json schema:@barterdude.consume_amqp(["queue1","queue2"],monitor,validation_schema={"$schema":"http://json-schema.org/draft-04/schema#","$id":"http://example.com/example.json","type":"object","title":"Message Schema","description":"The root schema comprises the entire JSON document.","additionalProperties":True,"required":["key"],"properties":{"key":{"$id":"#/properties/key","type":"string","title":"The Key Schema","description":"An explanation about message.","default":""}}},requeue_on_validation_fail=False# invalid messages are removed without requeue)You can still validate messages before produce them or when you want:frombarterdude.messageimportMessageValidationvalidator=MessageValidation(json_schema)validator.validate(message)Data ProtectionBarterdude takes in account GDPR data protection and by default doesn't log message body, but you can deactivate with environment variableBARTERDUDE_LOG_REDACTED=0Now messages body will be logged by Logging hook.This configuration just controls BarterDude's default Logging Hook and doesn't have effect on user custom user log. If you want to control your log with this configuration use:frombaterdude.confimportBARTERDUDE_LOG_REDACTEDHTTP EndpointsSimple endpointsIf you want to expose and HTTP endpoint, you can easily do that to your Barterdude worker by adding a route mapped to a hook.barterdude.add_endpoint(routes=["/some_hook"],methods=["GET"],hook=some_hook,)Barterdude's callback endpointYou can also expose an HTTP endpoint that calls the worker's callback to emulate a message being consumed and processed from a queue. This way you can make a request passing a body and header of a message and the response of this request will have all information of what the worker would do without really publishing the message.barterdude.add_callback_endpoint(routes=["/execute"],hook=execute,)In order to use a mock instance of the barterdude object, you also need to modify the signature of your callback method to receive a optional argument for the barterdude mock. Then you'll have to choose which one to use. Only the callback endpoint calls will pass the barterdude object to your callback.asyncdefexecute(rabbitmq_message:RabbitMQMessage,barterdude_arg=None):bd=barterdude_argifbarterdude_argisnotNoneelsebarterdudeRequest and response example:#Request{"body":{"list_id":105515152},"headers":{"trace_id":"random_id"}}#Response{"message_calls":[{"method":"accept","args":[],"kwargs":{}}],"barterdude_calls":[{"method":"publish_amqp","args":[],"kwargs":{"exchange":"NEXT_EXCHANGE_TO_BE_CALLED","data":{"list_id":1055151521,"subject":"vendo samsung galaxy s21","timestamp":1657231231000},"properties":{"headers":{"has_failed":false,"trace_id":"random_id"}}}}]}Side-effectsIf your callback has services with side-effects such as inserting a row in a database or updating an API, you can pass fake instances of these services that are going to be injected to prevent side-effects from happenning.barterdude.add_callback_endpoint(routes=["/execute"],hook=execute,mock_dependencies=[(fake_database_service,# fake service instance to be used by the worker"database_service",# name used in the data sharing/dependency injection),])Forcing side-effectsIf you want the message to be published when calling the callback endpoint, you can pass the parametershould_mock_barterdude: false. This way the message will be published. Also, you don't have to mock the services used by your worker, all side-effects will happen and you'll have your worker processing your message just like it would be when consuming from a queue.Request and response example:#Request{"body":{"list_id":105515152},"headers":{"trace_id":"random_id"},"should_mock_barterdude":false}#Response{"message_calls":[{"method":"accept","args":[],"kwargs":{}}]#messagewillbepublished,sowewon'thaveinformationaboutpublishmethod'scalls}TestingTo test async consumers we recommendasynctestlibfromasynctestimportTestCaseclassTestMain(TestCase):deftest_should_pass(self):self.assertTrue(True)We hope you enjoy! :wink:ContributeFor development and contributing, please followContributing GuideandALWAYSrespect theCode of Conduct