Package: django-bulk-update-or-create
django-bulk-update-or-createEveryone using Django ORM will eventually find himself doing batchupdate_or_createoperations: ingest files from external sources, sync with external APIs, etc.If the number of records is big, the slowliness ofQuerySet.update_or_createwill stand out: it is very practical to use but it always does oneSELECTand then oneINSERT(if select didn't return anything) orUPDATE/.save(if it did).Searching online shows that this does indeed happen to quite a few people though it doesn't seem to be any good solution:bulk_createis really fast if you know all records are new (and you're not using multi-table inheritance)bulk_updatedoes some nice voodoo to update several records with the sameUPDATEstatement (using a hugeWHEREcondition together withCASE), but you need to be sure they all existUPSERTs(INSERT .. ON DUPLICATE KEY UPDATE) look interesting (TODO on different package) but they will be retricted bybulk_createlimitations ==> cannot use on models with multi-table inheritanceThis package tries to tackle this introducingbulk_update_or_createto model QuerySet/Manager:update_or_create:(1 SELECT + 1 INSERT/UPDATE) * Nbulk_update_or_create:1 BIG_SELECT + 1 BIG_UPDATE + (lte_N) INSERTFor a batch of records:SELECTall from database (based on thematch_fieldparameter)Update records in memoryUsebulk_updatefor thoseUseINSERT/.createon each of the remainingThe (SOFTCORE)performance testlooks promising, more than 70% less time (average):$maketestcmd# default - sqliteDJANGO_SETTINGS_MODULE=settingstests/manage.pybulk_it
loopofupdate_or_create-allcreates:3.966486692428589
loopofupdate_or_create-allupdates:4.020653247833252
loopofupdate_or_create-halfhalf:3.9968857765197754
bulk_update_or_create-allcreates:2.949239730834961
bulk_update_or_create-allupdates:0.15633511543273926
bulk_update_or_create-halfhalf:1.4585723876953125# mysqlDJANGO_SETTINGS_MODULE=settings_mysqltests/manage.pybulk_it
loopofupdate_or_create-allcreates:5.511938571929932
loopofupdate_or_create-allupdates:5.321666955947876
loopofupdate_or_create-halfhalf:5.391834735870361
bulk_update_or_create-allcreates:1.5671980381011963
bulk_update_or_create-allupdates:0.14612770080566406
bulk_update_or_create-halfhalf:0.7262606620788574# postgresDJANGO_SETTINGS_MODULE=settings_postgresqltests/manage.pybulk_it
loopofupdate_or_create-allcreates:4.3584535121917725
loopofupdate_or_create-allupdates:3.6183276176452637
loopofupdate_or_create-halfhalf:4.145816087722778
bulk_update_or_create-allcreates:1.044851541519165
bulk_update_or_create-allupdates:0.14954638481140137
bulk_update_or_create-halfhalf:0.8407495021820068Installationpip install django-bulk-update-or-createAdd it to yourINSTALLED_APPSlist insettings.pyUsageuseBulkUpdateOrCreateQuerySetas manager of your model(s)fromdjango.dbimportmodelsfrombulk_update_or_createimportBulkUpdateOrCreateQuerySetclassRandomData(models.Model):objects=BulkUpdateOrCreateQuerySet.as_manager()uuid=models.IntegerField(unique=True)data=models.CharField(max_length=200,null=True,blank=True)callbulk_update_or_createitems=[RandomData(uuid=1,data='data for 1'),RandomData(uuid=2,data='data for 2'),]RandomData.objects.bulk_update_or_create(items,['data'],match_field='uuid')or use the context manager, if you are updating a big number of items, as it manages a batch queuewithRandomData.objects.bulk_update_or_create_context(['data'],match_field='uuid',batch_size=10)asbulkit:foriinrange(10000):bulkit.queue(RandomData(uuid=i,data=i+20))bulk_update_or_createsupportsyield_objects=Trueso you can iterate over the created/updated objects.bulk_update_or_create_contextprovides the same information to the callback function specified asstatus_cbDocsWIPToDoDocs!Add option to usebulk_createfor creates: assert model is not multi-table, if enabledFix the collation mess: the keyword argcase_insensitive_matchshould be dropped and collation detected in runtimeAdd support for multiplematch_field- probably will need to useWHERE (K1=X and K2=Y) or (K1=.. and K2 =..)instead ofINfor those, as that SQL standard doesn't seem widely adopted yetLink toUPSERTalternative package once done!