Package: django_pg_simple_hll
django_pg_simple_hllPostgres-only HyperLogLog cardinality aggregation for the Django ORM.Instead of counting the number of distinct users,%%timeSession.objects.aggregate(unique_users=Count("user_uuid",distinct=True))CPUtimes:user864µs,sys:2.46ms,total:3.32msWalltime:461ms{'unique_users':140000}you can approximate it,%%timeSession.objects.aggregate(approx_unique_users=HLLCardinality("user_uuid"))CPUtimes:user480µs,sys:1.54ms,total:2.02msWalltime:845ms{'approx_unique_users':144594}Or, instead of counting the number of distinct users per day,%%timelist(Session.objects.annotate(date_of_session=TruncDate("created")).values("date_of_session").annotate(unique_users=Count("user_uuid",distinct=True)).values("unique_users","date_of_session").order_by("date_of_session"))CPUtimes:user1.47ms,sys:3.16ms,total:4.64msWalltime:758ms[{'date_of_session':datetime.date(2023,4,28),'unique_users':20000},{'date_of_session':datetime.date(2023,4,29),'unique_users':40000},{'date_of_session':datetime.date(2023,4,30),'unique_users':60000},{'date_of_session':datetime.date(2023,5,1),'unique_users':80000},{'date_of_session':datetime.date(2023,5,2),'unique_users':100000},{'date_of_session':datetime.date(2023,5,3),'unique_users':120000},{'date_of_session':datetime.date(2023,5,4),'unique_users':140000}]you can approximate it,%%timelist(Session.objects.annotate(date_of_session=TruncDate("created")).values("date_of_session").annotate(approx_unique_users=HLLCardinality("user_uuid")).values("approx_unique_users","date_of_session").order_by("date_of_session"))CPUtimes:user1.53ms,sys:3.57ms,total:5.1msWalltime:838ms[{'date_of_session':datetime.date(2023,4,28),'approx_unique_users':19322},{'date_of_session':datetime.date(2023,4,29),'approx_unique_users':39356},{'date_of_session':datetime.date(2023,4,30),'approx_unique_users':61202},{'date_of_session':datetime.date(2023,5,1),'approx_unique_users':80917},{'date_of_session':datetime.date(2023,5,2),'approx_unique_users':102914},{'date_of_session':datetime.date(2023,5,3),'approx_unique_users':125637},{'date_of_session':datetime.date(2023,5,4),'approx_unique_users':144594}]The approximation is sometimes faster and uses less memory, particularly when faceting by other variables.By default, it uses a precision of 9, which tends to have an error of about 5%. You can set a higher precision (up to 26) through a second parameter:%%timelist(Session.objects.annotate(date_of_session=TruncDate("created")).values("date_of_session").annotate(approx_unique_users=HLLCardinality("user_uuid",11)).values("approx_unique_users","date_of_session").order_by("date_of_session"))CPUtimes:user1.71ms,sys:3.82ms,total:5.53msWalltime:1.98s[{'date_of_session':datetime.date(2023,4,28),'approx_unique_users':20013},{'date_of_session':datetime.date(2023,4,29),'approx_unique_users':39616},{'date_of_session':datetime.date(2023,4,30),'approx_unique_users':59312},{'date_of_session':datetime.date(2023,5,1),'approx_unique_users':81278},{'date_of_session':datetime.date(2023,5,2),'approx_unique_users':101880},{'date_of_session':datetime.date(2023,5,3),'approx_unique_users':122343},{'date_of_session':datetime.date(2023,5,4),'approx_unique_users':141375}]You can also set a lower precision (down to 4) for a faster, less accurate estimate.%%timelist(Session.objects.annotate(date_of_session=TruncDate("created")).values("date_of_session").annotate(approx_unique_users=HLLCardinality("user_uuid",7)).values("approx_unique_users","date_of_session").order_by("date_of_session"))CPUtimes:user1.3ms,sys:2.85ms,total:4.15msWalltime:547ms[{'date_of_session':datetime.date(2023,4,28),'approx_unique_users':19363},{'date_of_session':datetime.date(2023,4,29),'approx_unique_users':40627},{'date_of_session':datetime.date(2023,4,30),'approx_unique_users':63159},{'date_of_session':datetime.date(2023,5,1),'approx_unique_users':83371},{'date_of_session':datetime.date(2023,5,2),'approx_unique_users':96185},{'date_of_session':datetime.date(2023,5,3),'approx_unique_users':116565},{'date_of_session':datetime.date(2023,5,4),'approx_unique_users':143588}]HLL works by hashing each value it considers. For some reason, the aggregation is sometimes faster when that hashing is performed outside of the aggregation. So, it is also available in two separate steps:%%timelist(Session.objects.annotate(date_of_session=TruncDate("created")).values("date_of_session").annotate(approx_unique_users=HLLCardinalityFromHash(HLLHash("user_uuid"),9)).values("approx_unique_users","date_of_session").order_by("date_of_session"))CPUtimes:user1.66ms,sys:1.23ms,total:2.89msWalltime:662ms[{'date_of_session':datetime.date(2023,6,2),'approx_unique_users':19322},{'date_of_session':datetime.date(2023,6,3),'approx_unique_users':39356},{'date_of_session':datetime.date(2023,6,4),'approx_unique_users':61202},{'date_of_session':datetime.date(2023,6,5),'approx_unique_users':80917},{'date_of_session':datetime.date(2023,6,6),'approx_unique_users':102914},{'date_of_session':datetime.date(2023,6,7),'approx_unique_users':125637},{'date_of_session':datetime.date(2023,6,8),'approx_unique_users':144594}]The aggregation is also available in SQL for analytics:selectdate_trunc('day',created)asdate_of_session,hll_cardinality(user_uuid,11)asapprox_unique_usersfromtestapp_sessiongroupbydate_trunc('day',created)orderbydate_of_session;date_of_session|approx_unique_users------------------------+---------------------2023-04-2800:00:00+00|200132023-04-2900:00:00+00|396162023-04-3000:00:00+00|593122023-05-0100:00:00+00|812782023-05-0200:00:00+00|1018802023-05-0300:00:00+00|1223432023-05-0400:00:00+00|141375(7rows)Time:2121.662ms(00:02.122)orselectdate_trunc('day',created)asdate_of_session,hll_cardinality_from_hash(hll_hash(user_uuid),11)asapprox_unique_usersfromtestapp_sessiongroupbydate_trunc('day',created)orderbydate_of_session;date_of_session|approx_unique_users------------------------+---------------------2023-06-0200:00:00+00|200132023-06-0300:00:00+00|396162023-06-0400:00:00+00|593122023-06-0500:00:00+00|812782023-06-0600:00:00+00|1018802023-06-0700:00:00+00|1223432023-06-0800:00:00+00|141375(7rows)Time:1810.445ms(00:01.810)How to useInstall the package:pipinstalldjango-pg-simple-hllAdd it to your DjangoINSTALLED_APPS:INSTALLED_APPS=[..."django_pg_simple_hll",...]Run migrationsdjango-adminmigratedjango_pg_simple_hllShould I use this?If you can usean optimised version, you should use that. It will be faster - although I haven't done any benchmarks.Use this if you can't install extensions on your database, such as onAmazon RDS.SQL onlyDon't care about the Django functions, and just want to be able to run the SQL?
The entire implementation is in asingle SQL fileNotes on SQL implementationThis is a low-privilege implementation ofHyperloglogapproximate cardinality aggregation written in SQL and somePL/pgSQL. Read about ithere,Notes on PerformanceThis is not a scientific test, but here is an example of performance running on an M1 MacBook Pro, running Postgres 15 in Docker.For a dataset of 560k rows and 140k unique values a precision of 8 matches the speed ofCOUNT(DISTINCT ...):selectcount(user_uuid)fromtestapp_session;count--------560000(1row)Time:227.959msselectcount(distinctuser_uuid)asunique_usersfromtestapp_session;unique_users--------------140000(1row)Time:476.819msselecthll_cardinality_from_hash(hll_hash(user_uuid),8)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------139309(1row)Time:402.569msselecthll_cardinality_from_hash(hll_hash(user_uuid),11)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------141375(1row)Time:1193.696ms(00:01.194)For a dataset of 5.6M rows and 1.4M unique values:with a precision of 7,hll_aggregateruns in about 0.81x the speed ofCOUNT(DISTINCT ...)with a precision of 8, it roughly matches (0.94x)with a precision of 11, it runs in about 2.4xselectcount(user_uuid)fromtestapp_session;count---------5600000(1row)Time:3798.774ms(00:03.799)selectcount(distinctuser_uuid)asunique_usersfromtestapp_session;unique_users--------------1400000(1row)Time:4564.531ms(00:04.565)selecthll_cardinality_from_hash(hll_hash(user_uuid),7)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------1308235(1row)Time:3710.460ms(00:03.710)selecthll_cardinality_from_hash(hll_hash(user_uuid),8)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------1407331(1row)Time:4382.913ms(00:04.383)selecthll_cardinality_from_hash(hll_hash(user_uuid),11)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------1390983(1row)Time:11027.942ms(00:11.028)For a dataset of 56M rows and 14M unique values:with a precision of 8,hll_aggregateruns in about 0.48x the speed ofCOUNT(DISTINCT ...)with a precision of 9, it runs in about 0.63xwith a precision of 10, it runs in about 0.73xwith a precision of 11, it runs in about 1.07xselectcount(user_uuid)fromtestapp_session;count----------56000000(1row)Time:45861.085ms(00:45.861)selectcount(distinctuser_uuid)asunique_usersfromtestapp_session;unique_users--------------14000000(1row)Time:119551.361ms(01:59.551)selecthll_cardinality_from_hash(hll_hash(user_uuid),8)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------13469591(1row)Time:58370.988ms(00:58.371)selecthll_cardinality_from_hash(hll_hash(user_uuid),9)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------13028181(1row)Time:75231.592ms(01:15.232)selecthll_cardinality_from_hash(hll_hash(user_uuid),10)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------13147406(1row)Time:87497.737ms(01:27.498)selecthll_cardinality_from_hash(hll_hash(user_uuid),11)asapprox_unique_usersfromtestapp_session;approx_unique_users---------------------13363203(1row)Time:128338.223ms(02:08.338)The real advantage though is in combination with other properties:E.g. this query runs 0.37x quicker if usinghll_aggregatethan withCOUNT(DISTINCT ...):selectdate_trunc('day',created)asdate_of_session,count(distinctuser_uuid)asunique_usersfromtestapp_sessiongroupbydate_trunc('day',created)orderbydate_of_session;date_of_session|unique_users------------------------+--------------2023-06-0200:00:00+00|20000002023-06-0300:00:00+00|40000002023-06-0400:00:00+00|60000002023-06-0500:00:00+00|80000002023-06-0600:00:00+00|100000002023-06-0700:00:00+00|120000002023-06-0800:00:00+00|14000000(7rows)Time:157028.067ms(02:37.028)selectdate_trunc('day',created)asdate_of_session,hll_cardinality_from_hash(hll_hash(user_uuid),9)asunique_usersfromtestapp_sessiongroupbydate_trunc('day',created)orderbydate_of_session;date_of_session|unique_users------------------------+--------------2023-06-0200:00:00+00|19639722023-06-0300:00:00+00|40968762023-06-0400:00:00+00|58693392023-06-0500:00:00+00|74128432023-06-0600:00:00+00|94010102023-06-0700:00:00+00|114438362023-06-0800:00:00+00|13028181(7rows)Time:58778.194ms(00:58.778)