Package: django-pg-hll
django-pg-hllProvides a django wrapper forpostgresql-hll library by CitusDataRequirementsPython 3.5+django >= 1.9 (tested 2.2+)PostgreSQL 9.4+ (tested 9.6+)InstallationInstall via pip:pip install django-pg-hllor via setup.py:python setup.py installUsagePrerequisitesInstallpostgresql-hll extensionCreating hll extensionIf your user has super-admin privileges you can create Hll extension using migrations.
If you use django 1.10+ you can usedjango_pg_hll.migrations.HllExtensionin your migration file.
If you have older version you can use the following:migrations.RunSQL('CREATE EXTENSION IF NOT EXISTS hll;',reverse_sql='DROP EXTENSION hll;')Creating table with hll fieldAdd HllField to your model:fromdjango.dbimportmodelsfromdjango_pg_hllimportHllFieldclassMyModel(models.Model):hll=HllField()Callmakemigrationsto create a migrationCallmigrateto apply migration.Hll valuesIn order to create and update Hll this library introduces a set of functions
(corresponding topostgres-hll hash functions),
to hash values:fromdjango_pg_hllimportHllField# Empty hllHllEmpty()# Empty hll with custom configuration parameters# hll_empty([log2m[, regwidth[, expthresh[, sparseon]]]])HllEmpty(13,2,1,0)# Hash from booleanHllBoolean(True)# Hash from integer with different rangesHllSmallInt(1)HllInteger(65540)HllBigint(2147483650)# Hash from bytes sequenceHllByteA(b'test')# Hash from textHllText('test')# Auto detection of type by postgres-hllHllAny('some data')To save a value to HllField, you can pass any of these functions as a value:fromdjango_pg_hllimportHllIntegerinstance=MyModel.objects.create(hll=HllInteger(123))instance.hll|=HllInteger(456)instance.save()Chaining hll valuesHll values can be chained with each other and functions likedjango.db.models.Fusing|operator.The chaining result will bedjango_pg_hll.values.HllSetinstance, which can be also saved to database.You can also chain simple values and iterables.
In this case, library will try to detect appropriate hashing function, based on value.Important: Native django functions can't be used as chain start, as|operator is redeclared for HllValue instances.Example:fromdjango_pg_hllimportHllIntegerfromdjango.db.modelsimportFinstance=MyModel.objects.create(hll=HllInteger(123))# This worksinstance.hll|=HllInteger(456)instance.hll=HllInteger(456)|F('hll')instance.hll|=789# HllSmallInt will be usedinstance.hll|=100500# HllInteger will be usedinstance.hll|=True# HllBoolean will be usedinstance.hll|={1,2,3,4,5}# set. HllSmallInt will be used.# This throws exception, as F function doesn't support bitor operatorinstance.hll=F('hll')|HllInteger(456)Hashing seedYou can passhash_seedoptional argument to any HllValue, expecting data.Look herefor more details about hashing.Filtering QuerySetHllField realizes several lookups (returning float value) in order to make filtering easier:# EqualityMyModel.objects.filter(hll=HllInteger(1)).count()MyModel.objects.exclude(hll=HllInteger(2)).count()# CardinalityMyModel.objects.filter(hll__cardinality=3).count()# Configuration lookupsMyModel.objects.filter(hll__schema_version=1).count()MyModel.objects.filter(hll__type=1).count()MyModel.objects.filter(hll__log2m=11).count()MyModel.objects.filter(hll__regwidth=2).count()MyModel.objects.filter(hll__sparseon=1).count()Aggregate functionsIn order to count aggregations and annotations, library provides aggregate functions:django_pg_hll.aggregate.CardinalityCounts cardinality of hll fielddjango_pg_hll.aggregate.UnionAggAggregates multiple hll fields to one hll.django_pg_hll.aggregate.UnionAggCardinalityCounts cardinality of hll, combined by UnionAgg function. In fact, it doesCardinality(UnionAgg(hll)).P. s. django doesn't give ability to use function inside function.django_pg_hll.aggregate.CardinalitySumCounts sum of multiple rows hll cardinalities. In fact, it doesSum(Cardinality(hll)).P. s. django doesn't give ability to use function inside function.fromdjango.dbimportmodelsfromdjango_pg_hll.aggregateimportCardinality,UnionAggCardinality,CardinalitySumfromdjango_pg_hll.fieldsimportHllFieldfromdjango_pg_hll.valuesimportHllIntegerclassForeignModel(models.Model):passclassMyModel(models.Model):hll=HllField()fk=models.ForeignKey(ForeignModel)MyModel.objects.bulk_create([MyModel(fk=1,hll=HllInteger(1)),MyModel(fk=2,hll=HllInteger(2)|HllInteger(3)|HllInteger(4)),MyModel(fk=3,hll=HllInteger(4))])MyModel.objects.annotate(card=Cardinality('hll_field')).values_list('id','card')# outputs (1, 1), (2, 3), (3, 1)# Count cardinality for hll, build by union of all rows# 4 element exists in rows with fk=2 and fk=3. After union it gives single resultForeignModel.objects.annotate(card=UnionAggCardinality('testmodel__hll_field')).values_list('card',flat=True)# outputs [4]# Count sum of cardinalities for each rowForeignModel.objects.annotate(card=CardinalitySum('testmodel__hll_field')).values_list('card',flat=True)# outputs [5]Configuration aggregate functionsIn order to get hll field creation parameters, library provides aggregate functions:django_pg_hll.aggregate.HllSchemaVersionReturns the schema version value (integer) of the hlldjango_pg_hll.aggregate.HllTypeReturns the schema version-specific type value (integer) of the hll.
See thestorage specification (v1.0.0)for more details.django_pg_hll.aggregate.HllRegWidthReturns the register bit-width (integer) of the hlldjango_pg_hll.aggregate.HllLog2MReturns the log-base-2 of the number of registers of the hll.
If the hll is not of type FULL or SPARSE it returns the log2m value which would be used if the hll were promoted.django_pg_hll.aggregate.HllExpThresholdReturns an array with 2 elements of the specified and effective EXPLICIT promotion cutoffs for the hll.
The specified cutoff and the effective cutoff will be the same unless expthresh has been set to 'auto' (-1).
In that case the specified value will be -1 and the effective value will be the implementation-dependent number
of explicit values that will be stored before an EXPLICIT hll is promoted.django_pg_hll.aggregate.HllSParseOnReturns 1 if the SPARSE representation is enabled for the hll, and 0 otherwisefromdjango.dbimportmodelsfromdjango_pg_hll.aggregateimportHllLog2Mfromdjango_pg_hll.fieldsimportHllFieldfromdjango_pg_hll.valuesimportHllEmpty,HllIntegerclassMyModel(models.Model):default_hll=HllField()configured_hll=HllField(log2m=13,regwidth=2,expthresh=1,sparseon=0)MyModel.objects.create(fk=1,hll=HllInteger(1),configured_hll=HllEmpty(13,2,1,0))MyModel.objects.annotate(log2m=HllLog2M('default_hll'),log2m_conf=HllLog2M('configured_hll')).\values_list('log2m','log2m_conf')# outputs (11, 13)django-pg-bulk-updateintegrationThis library provides ahll_concatset function,
allowing to use hll inbulk_updateandbulk_update_or_createqueries.MyModel.objects.bulk_update_or_create([{'id':100501,'hll_field':HllInteger(1)},{'id':100502,'hll_field':HllInteger(2)|HllInteger(3)}],set_functions={'hll_field':'hll_concat'})Running testsRunning in dockerInstalldocker and docker-composeRundocker build . --tag django-pg-hllin project directoryRundocker-compose run run_testsin project directoryRunning in virtual environmentInstall all requirements listed aboveCreate virtual environmentCreate a superuser named 'test' on your local Postgres instance:CREATEROLEtest;ALTERROLEtestWITHSUPERUSER;ALTERROLEtestWITHLOGIN;ALTERROLEtestPASSWORD'test';CREATEDATABASEtestOWNERtest;Install requirementspip3 install -U -r requirements-test.txtStart testspython3 runtests.py