Package: django-clickhouse-backend
Django ClickHouse Database BackendDjango clickhouse backend is adjango database backendforclickhousedatabase. This project allows using django ORM to interact with
clickhouse, the goal of the project is to operate clickhouse like operating mysql, postgresql in django.Thanks toclickhouse driver, django clickhouse backend use it asDBAPI.
Thanks toclickhouse pool, it makes clickhouse connection pool.ReadDocumentationfor more.Features:Reuse most of the existed django ORM facilities, minimize your learning costs.Connect to clickhouse efficiently viaclickhouse native interfaceand connection pool.No other intermediate storage, no need to synchronize data, just interact directly with clickhouse.Support clickhouse specific schema features such asEngineandIndex.Support most types of table migrations.Support creating test database and table, working with django TestCase and pytest-django.Support most clickhouse data types.SupportSETTINGS in SELECT Query.Notes:Not tested upon all versions of clickhouse-server, clickhouse-server 22.x.y.z or over is suggested.Aggregation functions result in 0 or nan (Not NULL) when data set is empty. max/min/sum/count is 0, avg/STDDEV_POP/VAR_POP is nan.In outer join, clickhouse will set missing columns to empty values (0 for number, empty string for text, unix epoch for date/datatime) instead of NULL.
So Count("book") resolve to 1 in a missing LEFT OUTER JOIN match, not 0.
In aggregation expression Avg("book__rating", default=2.5), default=2.5 have no effect in a missing match.Clickhouse does not support unique constraint and foreignkey constraint.ForeignKey,ManyToManyFieldandOneToOneFieldcan be used with clickhouse backend, but no database level constraints will be added, so there could be some consistency problems.Clickhouse does not support transaction. If any exception occurs during migrating, then your clickhouse database will be in an untracked state. Any migration should be full tested in test environment before deployed to production environment.This project does not support migrations of changing table engine and settings yet.Requirements:Python>= 3.7Django>= 3.2clickhouse driverclickhouse poolGet startedInstallation$pipinstalldjango-clickhouse-backendor$gitclonehttps://github.com/jayvynl/django-clickhouse-backend
$cddjango-clickhouse-backend
$pythonsetup.pyinstallConfigurationOnlyENGINEis required in database setting, other options have default values.ENGINE: required, set toclickhouse_backend.backend.NAME: database name, defaultdefault.HOST: database host, defaultlocalhost.PORT: database port, default9000.USER: database user, defaultdefault.PASSWORD: database password, default empty.In the most cases, you may just use clickhouse to store some big events tables,
and use some RDBMS to store other tables.
Here I give an example setting for clickhouse and postgresql.INSTALLED_APPS=[# ..."clickhouse_backend",# ...]DATABASES={"default":{"ENGINE":"django.db.backends.postgresql","HOST":"localhost","USER":"postgres","PASSWORD":"123456","NAME":"postgres",},"clickhouse":{"ENGINE":"clickhouse_backend.backend","NAME":"default","HOST":"localhost","USER":"DB_USER","PASSWORD":"DB_PASSWORD",}}DATABASE_ROUTERS=["dbrouters.ClickHouseRouter"]# dbrouters.pyfromclickhouse_backend.modelsimportClickhouseModeldefget_subclasses(class_):classes=class_.__subclasses__()index=0whileindex<len(classes):classes.extend(classes[index].__subclasses__())index+=1returnlist(set(classes))classClickHouseRouter:def__init__(self):self.route_model_names=set()formodelinget_subclasses(ClickhouseModel):ifmodel._meta.abstract:continueself.route_model_names.add(model._meta.label_lower)defdb_for_read(self,model,**hints):if(model._meta.label_lowerinself.route_model_namesorhints.get("clickhouse")):return"clickhouse"returnNonedefdb_for_write(self,model,**hints):if(model._meta.label_lowerinself.route_model_namesorhints.get("clickhouse")):return"clickhouse"returnNonedefallow_migrate(self,db,app_label,model_name=None,**hints):if(f"{app_label}.{model_name}"inself.route_model_namesorhints.get("clickhouse")):returndb=="clickhouse"elifdb=="clickhouse":returnFalsereturnNoneYou should usedatabase routerto
automatically route your queries to the right database. In the preceding example, I write a database router which route all
queries from subclasses ofclickhouse_backend.models.ClickhouseModelor custom migrations with aclickhousehint key to clickhouse.
All other queries are routed to the default database (postgresql).Model DefinitionClickhouse backend support django builtin fields and clickhouse specific fields.Readfields documentationfor more.Notices about model definition:import models from clickhouse_backend, not from django.dbadd low_cardinality for StringFiled, when the data field cardinality is relatively low, this configuration can significantly improve query performancecannot use db_index=True in Field, but we can add in the Meta indexesneed to specify the ordering in Meta just for default query orderingneed to specify the engine for clickhouse, specify the order_by for clickhouse order and the partition_by argumentfromdjango.db.modelsimportCheckConstraint,IntegerChoices,Qfromdjango.utilsimporttimezonefromclickhouse_backendimportmodelsclassEvent(models.ClickhouseModel):classAction(IntegerChoices):PASS=1DROP=2ALERT=3ip=models.GenericIPAddressField(default="::")ipv4=models.IPv4Field(default="127.0.0.1")ip_nullable=models.GenericIPAddressField(null=True)port=models.UInt16Field(default=0)protocol=models.StringField(default="",low_cardinality=True)content=models.JSONField(default=dict)timestamp=models.DateTime64Field(default=timezone.now)created_at=models.DateTime64Field(auto_now_add=True)action=models.EnumField(choices=Action.choices,default=Action.PASS)classMeta:ordering=["-timestamp"]engine=models.MergeTree(primary_key="timestamp",order_by=("timestamp","id"),partition_by=models.toYYYYMMDD("timestamp"),index_granularity=1024,index_granularity_bytes=1<<20,enable_mixed_granularity_parts=1,)indexes=[models.Index(fields=["ip"],name="ip_set_idx",type=models.Set(1000),granularity=4),models.Index(fields=["ipv4"],name="ipv4_bloom_idx",type=models.BloomFilter(0.001),granularity=1)]constraints=(CheckConstraint(name="port_range",check=Q(port__gte=0,port__lte=65535),),)Migration$pythonmanage.pymakemigrationsthis operation will generate migration file under apps/migrations/then we mirgrate$pythonmanage.pymigrate--databaseclickhousefor the first time run, this operation will generate django_migrations table with create table sql like this>showcreatetabledjango_migrations;CREATETABLEother.django_migrations(`id`Int64,`app`FixedString(255),`name`FixedString(255),`applied`DateTime64(6,'UTC'))ENGINE=MergeTreeORDERBYidSETTINGSindex_granularity=8192we can query it with results like this>select*fromdjango_migrations;┌──────────────────id─┬─app─────┬─name─────────┬────────────────────applied─┐│1626937818115211264│testapp│0001_initial│2023-02-1813:32:57.538472│└─────────────────────┴─────────┴──────────────┴────────────────────────────┘migrate will create a table with name event as we define in the models>showcreatetableevent;CREATETABLEother.event(`id`Int64,`ip`IPv6,`ipv4`IPv6,`ip_nullable`Nullable(IPv6),`port`UInt16,`protocol`LowCardinality(String),`content`String,`timestamp`DateTime64(6,'UTC'),`created_at`DateTime64(6,'UTC'),`action`Enum8('Pass'=1,'Drop'=2,'Alert'=3),INDEXip_set_idxipTYPEset(1000)GRANULARITY4,INDEXport_bloom_idxportTYPEbloom_filter(0.001)GRANULARITY1,CONSTRAINTport_rangeCHECK(port>=0)AND(port<=65535))ENGINE=ReplacingMergeTreePARTITIONBYtoYYYYMMDD(timestamp)ORDERBYidSETTINGSindex_granularity=8192Operate Datacreateforiinrange(10):Event.objects.create(ip_nullable=None,port=i,protocol="HTTP",content="test",action=Event.Action.PASS.value)assertEvent.objects.count()==10queryqueryset=Event.objects.filter(content="test")foriinqueryset:print(i)updateEvent.objects.filter(port__in=[1,2,3]).update(protocol="TCP")time.sleep(1)assertEvent.objects.filter(protocol="TCP").count()==3deleteEvent.objects.filter(protocol="TCP").delete()time.sleep(1)assertnotEvent.objects.filter(protocol="TCP").exists()Except for the model definition, all other operations are like operating relational databases such as mysql and postgresqlTestingWriting testcase is all the same as normal django project. You can use django TestCase or pytest-django.Notice:clickhouse use mutations fordeleting or updating.
By default, data mutations is processed asynchronously.
That is, when you update or delete a row, clickhouse will perform the action after a period of time.
So you should change this default behavior in testing for deleting or updating.
There are 2 ways to do that:Config database engine as follows, this setsmutations_sync=1at session scope.DATABASES={"default":{"ENGINE":"clickhouse_backend.backend","OPTIONS":{"settings":{"mutations_sync":1,}}}}UseSETTINGS in SELECT Query.Event.objects.filter(protocol="UDP").settings(mutations_sync=1).delete()Sample test case.fromdjango.testimportTestCaseclassTestEvent(TestCase):deftest_spam(self):assertEvent.objects.count()==0Distributed tableThis backend supportdistributed DDL queries (ON CLUSTER clause)anddistributed table engine.The following example assumes that a cluster defined bydocker compose in this repositoryis used.
This cluster name iscluster, it has 2 shards, every shard has 2 replica.ConfigurationDATABASES={"default":{"ENGINE":"clickhouse_backend.backend","OPTIONS":{"migration_cluster":"cluster","settings":{"mutations_sync":2,"insert_distributed_sync":1,"insert_quorum":2,"alter_sync":2,},},"TEST":{"cluster":"cluster"},},"s1r2":{"ENGINE":"clickhouse_backend.backend","PORT":9001,"OPTIONS":{"migration_cluster":"cluster","settings":{"mutations_sync":2,"insert_distributed_sync":1,"insert_quorum":2,"alter_sync":2,},},"TEST":{"cluster":"cluster","managed":False,"DEPENDENCIES":["default"]},},"s2r1":{"ENGINE":"clickhouse_backend.backend","PORT":9002,"OPTIONS":{"migration_cluster":"cluster","settings":{"mutations_sync":2,"insert_distributed_sync":1,"insert_quorum":2,"alter_sync":2,},},"TEST":{"cluster":"cluster","managed":False,"DEPENDENCIES":["default"]},},"s2r2":{"ENGINE":"clickhouse_backend.backend","PORT":9003,"OPTIONS":{"migration_cluster":"cluster","settings":{"mutations_sync":2,"insert_distributed_sync":1,"insert_quorum":2,"alter_sync":2,},},"TEST":{"cluster":"cluster","managed":False,"DEPENDENCIES":["default"]},},}Extra settings explanation:"migration_cluster": "cluster"Migration table will be created on this cluster if this setting is specified, otherwise only local migration table is created."mutations_sync": 2This is suggested if you want to testdata mutationson replicated table.Don'tset this in production environment."insert_distributed_sync": 1This is suggested if you want to test inserting data into distributed table.Don'tset this in production environment."insert_quorum": 2This is suggested if you want to test inserting data into replicated table.
The value is set to replica number."alter_sync": 2This is suggested if you want to test altering or truncating replicated table.Don'tset this in production environment."TEST": {"cluster": "cluster", "managed": False, "DEPENDENCIES": ["default"]}Test database will be created on this cluster.
If you have multiple database connections to the same cluster and want to run tests over all these connections,
then only one connection should set"managed": True(the default value), other connections should set"managed": False.
So that test database will not be created multiple times.If your managed database alias iss1r2insteaddefault,"DEPENDENCIES": ["s1r2"]should be set to ensure thecreation order for test databases.Do not hardcode database name when you define replicated table or distributed table.
Because test database name is different from deployed database name.ModelclusterinMetaclass will make models being created on cluster.fromclickhouse_backendimportmodelsclassStudent(models.ClickhouseModel):name=models.StringField()address=models.StringField()score=models.Int8Field()classMeta:engine=models.ReplicatedMergeTree("/clickhouse/tables/{uuid}/{shard}",# Or if you want to use database name or table name, you should also use macro instead of hardcoded name.# "/clickhouse/tables/{database}/{table}/{shard}","{replica}",order_by="id")cluster="cluster"classDistributedStudent(models.ClickhouseModel):name=models.StringField()score=models.Int8Field()classMeta:engine=models.Distributed("cluster",models.currentDatabase(),Student._meta.db_table,models.Rand())cluster="cluster"CRUDJust like normal table, you can do whatever you like to distributed table.students=DistributedStudent.objects.bulk_create([DistributedStudent(name=f"Student{i}",score=i*10)foriinrange(10)])assertDistributedStudent.objects.count()==10DistributedStudent.objects.filter(id__in=[s.idforsinstudents[5:]]).update(name="lol")DistributedStudent.objects.filter(id__in=[s.idforsinstudents[:5]]).delete()MigrateIfmigration_clusteris not specified in database configuration. You should always run migrating on one specific cluster node.
Because other nodes do not know whether migrations have been applied by any other node.Ifmigration_clusteris specified. Then migration table(nameddjango_migrations) will be created on the specified cluster.
When applied,migration operationsof model with cluster defined inMetaclass
will be executed on cluster, other migration operations will be executed locally.
This means distributed table will be created on all nodes as long as any node has applied the migrations.
Other local table will only be created on node which has applied the migrations.If you want to use local table in all nodes, you should apply migrations multiple times on all nodes.
But remember, these local tables store data separately, currently this backend do not provide means to query data from other nodes.pythonmanage.pymigrate
pythonmanage.pymigrate--databases1r2
pythonmanage.pymigrate--databases2r1
pythonmanage.pymigrate--databases2r2UpdateWhen updated from django clickhouse backend 1.1.0 or lower, you should not add cluster related settings to your
existing project. Because:Migration table schema won't be changed if you add or removemigration_on_cluster. Andpython mange.py migratewill work abnormally.If you add cluster to your existing model'sMetaclass, no schema changes will occur, this project does not support this yet.If you really want to use cluster feature with existing project, you should manage schema changes yourself.
These steps should be tested carefully in test environment.Clickhouse docsmay be helpful.Apply all your existing migrations.Change your settings and model.Generate new migrations.Log into your clickhouse database and change table schemas to reflect your models.Apply migrations with fake flag.pythonmanage.pymigrate# Change your settings and modelpythonmanage.pymakemigrations# Log into your clickhouse database and change table schemas to reflect your models.pythonmanage.pymigrate--fakeTestTo run test for this project:$gitclonehttps://github.com/jayvynl/django-clickhouse-backend
$cddjango-clickhouse-backend# docker and docker-compose are required.$docker-composeup-d
$pythontests/runtests.py# run test for every python version and django version$pipinstalltox
$toxChangelogAll changelogs.ContributingReadContributing guide.LicenseDjango clickhouse backend is distributed under theMIT license.