Package: django-cqrs-enkonix
Django CQRSdjango-cqrsis an Django application, that implements CQRS data synchronisation between several Django microservices.CQRSIn Connect we have a rather complex Domain Model. There are many microservices, that aredecomposed by subdomainand which followdatabase-per-servicepattern. These microservices have rich and consistent APIs. They are deployed in cloud k8s cluster and scale automatically under load. Many of these services aggregate data from other ones and usuallyAPI Compositionis totally enough. But, some services are working too slowly with API JOINS, so another pattern needs to be applied.The pattern, that solves this issue is calledCQRS - Command Query Responsibility Segregation. Core idea behind this pattern is that view databases (replicas) are defined for efficient querying and DB joins. Applications keep their replicas up to data by subscribing toDomain eventspublished by the service that owns the data. Data iseventually consistentand that's okay for non-critical business transactions.ExamplesIntegrationSetupRabbitMQInstalldjango-cqrsApply changes to master service, according to RabbitMQ settings# models.pyfromdjango.dbimportmodelsfromdj_cqrs.mixinsimportMasterMixin,RawMasterMixinclassAccount(MasterMixin,models.Model):CQRS_ID='account'CQRS_PRODUCE=True# set this to False to prevent sending instances to TransportclassAuthor(MasterMixin,models.Model):CQRS_ID='author'CQRS_SERIALIZER='app.api.AuthorSerializer'# For cases of Diamond Multiinheritance the following approach could be used:frommptt.modelsimportMPTTModelfromdj_cqrs.metasimportMasterMetaclassComplexInheritanceModel(MPTTModel,RawMasterMixin):passMasterMeta.register(ComplexInheritanceModel)# settings.pyCQRS={'transport':'dj_cqrs.transport.rabbit_mq.RabbitMQTransport','host':RABBITMQ_HOST,'port':RABBITMQ_PORT,'user':RABBITMQ_USERNAME,'password':RABBITMQ_PASSWORD,}Apply changes to replica service, according to RabbitMQ settingsfromdjango.dbimportmodelsfromdj_cqrs.mixinsimportReplicaMixinclassAccountRef(ReplicaMixin,models.Model):CQRS_ID='account'id=models.IntegerField(primary_key=True)classAuthorRef(ReplicaMixin,models.Model):CQRS_ID='author'CQRS_CUSTOM_SERIALIZATION=True@classmethoddefcqrs_create(cls,sync,**mapped_data):# Override herepassdefcqrs_update(self,sync,**mapped_data):# Override herepass# settings.pyCQRS={'transport':'dj_cqrs.transport.RabbitMQTransport','queue':'account_replica','host':RABBITMQ_HOST,'port':RABBITMQ_PORT,'user':RABBITMQ_USERNAME,'password':RABBITMQ_PASSWORD,}Apply migrations on both servicesRun consumer worker on replica service. Management command:python manage.py cqrs_consume -w 2NotesWhen there are master models with related entities in CQRS_SERIALIZER, it's important to have operations within atomic transactions.
CQRS sync will happen on transaction commit. Please, avoid saving master model within transaction more then once to reduce syncing and potential racing on replica side.
Updating of related model won't trigger CQRS automatic synchronization for master model. This needs to be done manually.Example:withtransaction.atomic():publisher=models.Publisher.objects.create(id=1,name='publisher')author=models.Author.objects.create(id=1,name='author',publisher=publisher)withtransaction.atomic():publisher.name='new'publisher.save()author.save()When only needed instances need to be synchronized, there is a methodis_sync_instanceto set filtering rule.
It's important to understand, that CQRS counting works even without syncing and rule is applied every time model is updated.Example:classFilteredSimplestModel(MasterMixin,models.Model):CQRS_ID='filter'name=models.CharField(max_length=200)defis_sync_instance(self):returnlen(str(self.name))>2UtilitiesBulk synchronizer without transport (usage example: it may be used for initial configuration). May be used at planned downtime.On master service:python manage.py cqrs_bulk_dump --cqrs-id=author->author.dumpOn replica service:python manage.py cqrs_bulk_load -i=author.dumpFilter synchronizer over transport (usage example: sync some specific records to a given replica). Can be used dynamically.To sync all replicas:python manage.py cqrs_sync --cqrs-id=author -f={"id__in": [1, 2]}To sync all instances only with one replica:python manage.py cqrs_sync --cqrs-id=author -f={} -q=replicaSet of diff synchronization tools ()To get diff and synchronize master service with replica service in K8S:kubectlexec-iMASTER_CONTAINER--pythonmanage.pycqrs_diff_master--cqrs-id=author|kubectlexec-iREPLICA_CONTAINER--pythonmanage.pycqrs_diff_replica|kubectlexec-iMASTER_CONTAINER--pythonmanage.pycqrs_diff_syncDevelopmentPython 3.5+Install dependenciesrequirements/dev.txtTestingUnit testingPython 3.5+Install dependenciesrequirements/test.txtexport PYTHONPATH=/your/path/to/django-cqrs/Check code style:flake8Run tests:pytestTests reports are generated intests/reports.out.xml- JUnit test resultscoverage.xml- Coverage xml resultsTo generate HTML coverage reports use:--cov-report html:tests/reports/cov_htmlIntegrational testingdocker-composecd integration_testsdocker-compose run master