Package: django-cqrs
Django CQRSdjango-cqrsis an Django application, that implements CQRS data synchronisation between several Django microservices.CQRSIn Connect we have a rather complex Domain Model. There are many microservices, that aredecomposed by subdomainand which followdatabase-per-servicepattern. These microservices have rich and consistent APIs. They are deployed in cloud k8s cluster and scale automatically under load. Many of these services aggregate data from other ones and usuallyAPI Compositionis totally enough. But, some services are working too slowly with API JOINS, so another pattern needs to be applied.The pattern, that solves this issue is calledCQRS - Command Query Responsibility Segregation. Core idea behind this pattern is that view databases (replicas) are defined for efficient querying and DB joins. Applications keep their replicas up to data by subscribing toDomain eventspublished by the service that owns the data. Data iseventually consistentand that's okay for non-critical business transactions.DocumentationFull documentation is available athttps://django-cqrs.readthedocs.org.ExamplesYou can find an example projecthereIntegrationSetupRabbitMQInstalldjango-cqrsApply changes to master service, according to RabbitMQ settings# models.pyfromdjango.dbimportmodelsfromdj_cqrs.mixinsimportMasterMixin,RawMasterMixinclassAccount(MasterMixin,models.Model):CQRS_ID='account'CQRS_PRODUCE=True# set this to False to prevent sending instances to TransportclassAuthor(MasterMixin,models.Model):CQRS_ID='author'CQRS_SERIALIZER='app.api.AuthorSerializer'# For cases of Diamond Multi-inheritance or in case of Proxy Django-models the following approach could be used:frommptt.modelsimportMPTTModelfromdj_cqrs.metasimportMasterMetaclassComplexInheritanceModel(MPTTModel,RawMasterMixin):CQRS_ID='diamond'classBaseModel(RawMasterMixin):CQRS_ID='base'classProxyModel(BaseModel):classMeta:proxy=TrueMasterMeta.register(ComplexInheritanceModel)MasterMeta.register(BaseModel)# settings.pyCQRS={'transport':'dj_cqrs.transport.rabbit_mq.RabbitMQTransport','host':RABBITMQ_HOST,'port':RABBITMQ_PORT,'user':RABBITMQ_USERNAME,'password':RABBITMQ_PASSWORD,}Apply changes to replica service, according to RabbitMQ settingsfromdjango.dbimportmodelsfromdj_cqrs.mixinsimportReplicaMixinclassAccountRef(ReplicaMixin,models.Model):CQRS_ID='account'id=models.IntegerField(primary_key=True)classAuthorRef(ReplicaMixin,models.Model):CQRS_ID='author'CQRS_CUSTOM_SERIALIZATION=True@classmethoddefcqrs_create(cls,sync,mapped_data,previous_data=None,meta=None):# Override herepassdefcqrs_update(self,sync,mapped_data,previous_data=None,meta=None):# Override herepass# settings.pyCQRS={'transport':'dj_cqrs.transport.RabbitMQTransport','queue':'account_replica','host':RABBITMQ_HOST,'port':RABBITMQ_PORT,'user':RABBITMQ_USERNAME,'password':RABBITMQ_PASSWORD,}Apply migrations on both servicesRun consumer worker on replica service. Management command:python manage.py cqrs_consume -w 2NotesWhen there are master models with related entities in CQRS_SERIALIZER, it's important to have operations within atomic transactions. CQRS sync will happen on transaction commit.Please, avoid saving different instances of the same entity within transaction to reduce syncing and potential racing on replica side.Updating of related model won't trigger CQRS automatic synchronization for master model. This needs to be done manually.By defaultupdate_fieldsdoesn't trigger CQRS logic, but it can be overridden for the whole application in settings:settings.CQRS={...'master':{'CQRS_AUTO_UPDATE_FIELDS':True,},...}or a special flag can be used in each place, where it's required to trigger CQRS flow:instance.save(update_fields=['name'],update_cqrs_fields=True)When only needed instances need to be synchronized, there is a methodis_sync_instanceto set filtering rule.
It's important to understand, that CQRS counting works even without syncing and rule is applied every time model is updated.Example:classFilteredSimplestModel(MasterMixin,models.Model):CQRS_ID='filter'name=models.CharField(max_length=200)defis_sync_instance(self):returnlen(str(self.name))>2Django AdminAdd action to synchronize master items from Django Admin page.fromdjango.dbimportmodelsfromdjango.contribimportadminfromdj_cqrs.admin_mixinsimportCQRSAdminMasterSyncMixinclassAccountAdmin(CQRSAdminMasterSyncMixin,admin.ModelAdmin):...admin.site.register(models.Account,AccountAdmin)If necessary, override_cqrs_sync_querysetfromCQRSAdminMasterSyncMixinto adjust the QuerySet and use it for synchronization.UtilitiesBulk synchronizer without transport (usage example: it may be used for initial configuration). May be used at planned downtime.On master service:python manage.py cqrs_bulk_dump --cqrs-id=author->author.dumpOn replica service:python manage.py cqrs_bulk_load -i=author.dumpFilter synchronizer over transport (usage example: sync some specific records to a given replica). Can be used dynamically.To sync all replicas:python manage.py cqrs_sync --cqrs-id=author -f={"id__in": [1, 2]}To sync all instances only with one replica:python manage.py cqrs_sync --cqrs-id=author -f={} -q=replicaSet of diff synchronization tools:To get diff and synchronize master service with replica service in K8S:kubectlexec-iMASTER_CONTAINER--pythonmanage.pycqrs_diff_master--cqrs-id=author|kubectlexec-iREPLICA_CONTAINER--pythonmanage.pycqrs_diff_replica|kubectlexec-iMASTER_CONTAINER--pythonmanage.pycqrs_diff_syncIf it's important to check sync and clean up deleted objects within replica service in K8S:kubectlexec-iREPLICA_CONTAINER--pythonmanage.pycqrs_deleted_diff_replica--cqrs-id=author|kubectlexec-iMASTER_CONTAINER--pythonmanage.pycqrs_deleted_diff_master|kubectlexec-iREPLICA_CONTAINER--pythonmanage.pycqrs_deleted_sync_replicaDevelopmentPython >= 3.8Install dependenciesrequirements/dev.txtWe useisortlibrary to order and format our imports, andblack- to format the code.
We check it usingflake8-isortandflake8-blacklibraries (automatically onflake8run).For convenience you may runisort . && black .to format the code.TestingUnit testingPython >= 3.8Install dependenciesrequirements/test.txtexport PYTHONPATH=/your/path/to/django-cqrs/Run tests with various RDBMS:cd integration_testsDB=postgres docker-compose -f docker-compose.yml -f rdbms.yml run app_testDB=mysql docker-compose -f docker-compose.yml -f rdbms.yml run app_testCheck code style:flake8Run tests:pytestTests reports are generated intests/reports.out.xml- JUnit test resultscoverage.xml- Coverage xml resultsTo generate HTML coverage reports use:--cov-report html:tests/reports/cov_htmlIntegrational testingdocker-composecd integration_testsdocker-compose run master