Package: django-pg-zero-downtime-migrations
django-pg-zero-downtime-migrationsDjango postgresql backend that apply migrations with respect to database locks.Installationpip install django-pg-zero-downtime-migrationsUsageTo enable zero downtime migrations for postgres just setup django backend provided by this package and add most safe settings:DATABASES = {
    'default': {
        'ENGINE': 'django_zero_downtime_migrations.backends.postgres',
        #'ENGINE': 'django_zero_downtime_migrations.backends.postgis',
        ...
    }
}
ZERO_DOWNTIME_MIGRATIONS_LOCK_TIMEOUT = '2s'
ZERO_DOWNTIME_MIGRATIONS_STATEMENT_TIMEOUT = '2s'
ZERO_DOWNTIME_MIGRATIONS_FLEXIBLE_STATEMENT_TIMEOUT = True
ZERO_DOWNTIME_MIGRATIONS_RAISE_FOR_UNSAFE = TrueNOTE:this backend brings zero downtime improvements only for migrations (schema andRunSQLoperations, but not forRunPythonoperation), for other purpose it works the same as standard django backend.NOTE:this package is in beta, please check your migrations SQL before applying on production and submit issue for any question.Differences with standard django backendThis backend provides same result state, but different way and with additional guarantees for avoiding stuck table locks.This backend doesn't use transactions for migrations (exceptRunPythonoperation), because not all SQL fixes can be run in transaction and it allows to avoid deadlocks for complex migration. So when your migration will down in middle of transaction you need fix it manually (instead potential downtime). For that reason good practice to make migration modules small as possible.Deployment flowThere are requirements for zero downtime deployment:We have one database;We have several instances with application - application always should be available, even you restart one of instances;We have balancer before instances;Our application works fine before, on and after migration - old application works fine with old and new database schema version;Our application works fine before, on and after instance updating - old and new application versions work fine with new database schema version.Flow:apply migrationsdisconnect instance form balancer, restart it and back to balancer - repeat this operation one by one for all instancesIf our deployment don't satisfy zero downtime deployment rules, then we split it to smaller deployments.SettingsZERO_DOWNTIME_MIGRATIONS_LOCK_TIMEOUTApplylock_timeoutfor SQL statements that requireACCESS EXCLUSIVElock, defaultNone:ZERO_DOWNTIME_MIGRATIONS_LOCK_TIMEOUT = '2s'Allowed values:None- current postgres setting usedother - timeout will be applied,0and equivalents mean that timeout will be disabledZERO_DOWNTIME_MIGRATIONS_STATEMENT_TIMEOUTApplystatement_timeoutfor SQL statements that requireACCESS EXCLUSIVElock, defaultNone:ZERO_DOWNTIME_MIGRATIONS_STATEMENT_TIMEOUT = '2s'Allowed values:None- current postgres setting usedother - timeout will be applied,0and equivalents mean that timeout will be disabledZERO_DOWNTIME_MIGRATIONS_FLEXIBLE_STATEMENT_TIMEOUTSetstatement_timeoutto0msfor SQL statements that requireSHARE UPDATE EXCLUSIVElock that useful in case whenstatement_timeoutenabled globally and you try run long-running operations like index creation or constraint validation, defaultFalse:ZERO_DOWNTIME_MIGRATIONS_FLEXIBLE_STATEMENT_TIMEOUT = TrueZERO_DOWNTIME_MIGRATIONS_RAISE_FOR_UNSAFEEnabled option doesn't allow run potential unsafe migration, defaultFalse:ZERO_DOWNTIME_MIGRATIONS_RAISE_FOR_UNSAFE = TrueZERO_DOWNTIME_DEFERRED_SQLDefine way to apply deferred sql, defaultTrue:ZERO_DOWNTIME_DEFERRED_SQL = TrueAllowed values:True- run deferred sql similar to default django way.False- run deferred sql as soon as possible.How it worksPostgres table level locksPostgres has different locks on table level that can conflict with each otherhttps://www.postgresql.org/docs/current/static/explicit-locking.html#LOCKING-TABLES:ACCESS SHAREROW SHAREROW EXCLUSIVESHARE UPDATE EXCLUSIVESHARESHARE ROW EXCLUSIVEEXCLUSIVEACCESS EXCLUSIVEACCESS SHAREXROW SHAREXXROW EXCLUSIVEXXXXSHARE UPDATE EXCLUSIVEXXXXXSHAREXXXXXSHARE ROW EXCLUSIVEXXXXXXEXCLUSIVEXXXXXXXACCESS EXCLUSIVEXXXXXXXXMigration and business logic locksLets split this lock to migration and business logic operations.Migration operations work synchronously in one thread and cover schema migrations (data migrations conflict with business logic operations same as business logic conflict concurrently).Business logic operations work concurrently.Migration lockslockoperationsACCESS EXCLUSIVECREATE SEQUENCE,DROP SEQUENCE,CREATE TABLE,DROP TABLE*,ALTER TABLE**,DROP INDEXSHARECREATE INDEXSHARE UPDATE EXCLUSIVECREATE INDEX CONCURRENTLY,DROP INDEX CONCURRENTLY,ALTER TABLE VALIDATE CONSTRAINT****:CREATE SEQUENCE,DROP SEQUENCE,CREATE TABLE,DROP TABLEshouldn't have conflicts, because your business logic shouldn't yet operate with created tables and shouldn't already operate with deleted tables.**: Not allALTER TABLEoperations takeACCESS EXCLUSIVElock, but all current django's migrations take ithttps://github.com/django/django/blob/master/django/db/backends/base/schema.py,https://github.com/django/django/blob/master/django/db/backends/postgresql/schema.pyandhttps://www.postgresql.org/docs/current/static/sql-altertable.html.***: Django doesn't haveVALIDATE CONSTRAINTlogic, but we will use it for some cases.Business logic lockslockoperationsconflict with lockconflict with operationsACCESS SHARESELECTACCESS EXCLUSIVEALTER TABLE,DROP INDEXROW SHARESELECT FOR UPDATEACCESS EXCLUSIVE,EXCLUSIVEALTER TABLE,DROP INDEXROW EXCLUSIVEINSERT,UPDATE,DELETEACCESS EXCLUSIVE,EXCLUSIVE,SHARE ROW EXCLUSIVE,SHAREALTER TABLE,DROP INDEX,CREATE INDEXSo you can find that all django schema changes for exist table conflicts with business logic, but fortunately they are safe or has safe alternative in general.Postgres row level locksAs business logic mostly works with table rows it's also important to understand lock conflicts on row levelhttps://www.postgresql.org/docs/current/static/explicit-locking.html#LOCKING-ROWS:lockFOR KEY SHAREFOR SHAREFOR NO KEY UPDATEFOR UPDATEFOR KEY SHAREXFOR SHAREXXFOR NO KEY UPDATEXXXFOR UPDATEXXXXMain point there is if you have two transactions that update one row, then second transaction will wait until first will be completed. So for business logic and data migrations better to avoid updates for whole table and use batch operations instead.NOTE:batch operations also can work faster because postgres can use more optimal execution plan with indexes for small data range.Transactions FIFO waitingFound same diagram in interesting articlehttp://pankrat.github.io/2015/django-migrations-without-downtimes/.In this diagram we can extract several metrics:operation time - time spent changing schema, in the case of long running operations on many rows tables likeCREATE INDEXorALTER TABLE ADD COLUMN SET DEFAULT, so you need a safe equivalent.waiting time - your migration will wait until all transactions complete, so there is issue for long running operations/transactions like analytic, so you need avoid it or disable during migration.queries per second + execution time and connections pool - if executing many queries, especially long running ones, they can consume all available database connections until the lock is released, so you need different optimizations there: run migrations when least busy, decrease query count and execution time, split data.too many operations in one transaction - you have issues in all previous points for one operation so if you have many operations in one transaction then you have more likelihood to get this issue, so you need avoid too many simultaneous operations in a single transaction (or even not run it in a transaction at all but being careful when an operation fails).Dealing with timeoutsPostgres has two settings to dealing withwaiting timeandoperation timepresented in diagram:lock_timeoutandstatement_timeout.SET lock_timeout TO '2s'allow you to avoid downtime when you have long running query/transaction before run migration (https://www.postgresql.org/docs/current/static/runtime-config-client.html#GUC-LOCK-TIMEOUT).SET statement_timeout TO '2s'allow you to avoid downtime when you have long running migration query (https://www.postgresql.org/docs/current/static/runtime-config-client.html#GUC-STATEMENT-TIMEOUT).DeadlocksThere no downtime issues for deadlocks, but too many operations in one transaction can take most conflicted lock and release it only after transaction commit or rollback. So it's a good idea to avoidACCESS EXCLUSIVElock operations and long time operations in one transaction. Deadlocks also can make you migration stuck on production deployment when different tables will be locked, for example, for FOREIGN KEY that takeACCESS EXCLUSIVElock for two tables.Rows and values storingPostgres store values of different types different ways. If you try to convert one type to another and it stored different way then postgres will rewrite all values. Fortunately some types stored same way and postgres need to do nothing to change type, but in some cases postgres need to check that all values have same with new type limitations, for example string length.Multiversion Concurrency ControlRegarding documentationhttps://www.postgresql.org/docs/current/static/mvcc-intro.htmldata consistency in postgres is maintained by using a multiversion model. This means that each SQL statement sees a snapshot of data. It has advantage for adding and deleting columns without any indexes, constrains and defaults do not change exist data, new version of data will be created onINSERTandUPDATE, delete just mark you record expired. All garbage will be collected later byVACUUMorAUTO VACUUM.Django migrations hacksAny schema changes can be processed with creation of new table and copy data to it, but it can take significant time.#namesafesafe alternativedescription1CREATE SEQUENCEXsafe operation, because your business logic shouldn't operate with new sequence on migration time *2DROP SEQUENCEXsafe operation, because your business logic shouldn't operate with this sequence on migration time *3CREATE TABLEXsafe operation, because your business logic shouldn't operate with new table on migration time *4DROP TABLEXsafe operation, because your business logic shouldn't operate with this table on migration time *5ALTER TABLE RENAME TOadd new table and copy dataunsafe operation, it's too hard write business logic that operate with two tables simultaneously, so proposeCREATE TABLEand then copy all data to new table *6ALTER TABLE SET TABLESPACEadd new table and copy dataunsafe operation, but probably you don't need it at all or often *7ALTER TABLE ADD COLUMNXsafe operation if withoutSET NOT NULL,SET DEFAULT,PRIMARY KEY,UNIQUE*8ALTER TABLE ADD COLUMN SET DEFAULTadd column and set defaultunsafe operation, because you spend time in migration to populate all values in table, so proposeALTER TABLE ADD COLUMNand then populate column and thenSET DEFAULT*9ALTER TABLE ADD COLUMN SET NOT NULL+/-unsafe operation, because doesn't work withoutSET DEFAULTor after migration old code can insert rows without new column and raise exception, so proposeALTER TABLE ADD COLUMNand then populate column and thenALTER TABLE ALTER COLUMN SET NOT NULL* and **10ALTER TABLE ADD COLUMN PRIMARY KEYadd index and add constraintunsafe operation, because you spend time in migration toCREATE INDEX, so proposeALTER TABLE ADD COLUMNand thenCREATE INDEX CONCURRENTLYand thenALTER TABLE ADD CONSTRAINT PRIMARY KEY USING INDEX***11ALTER TABLE ADD COLUMN UNIQUEadd index and add constraintunsafe operation, because you spend time in migration toCREATE INDEX, so proposeALTER TABLE ADD COLUMNand thenCREATE INDEX CONCURRENTLYand thenALTER TABLE ADD CONSTRAINT UNIQUE USING INDEX***12ALTER TABLE ALTER COLUMN TYPE+/-unsafe operation, because you spend time in migration to check that all items in column valid or to change type, but some operations can be safe ****13ALTER TABLE ALTER COLUMN SET NOT NULLadd check constraint beforeunsafe operation, because you spend time in migration to check that all items in columnNOT NULL, so proposeALTER TABLE ADD CONSTRAINT CHECKand thenALTER TABLE VALIDATE CONSTRAINTand thenALTER TABLE ALTER COLUMN SET NOT NULL**14ALTER TABLE ALTER COLUMN DROP NOT NULLXsafe operation15ALTER TABLE ALTER COLUMN SET DEFAULTXsafe operation16ALTER TABLE ALTER COLUMN DROP DEFAULTXsafe operation17ALTER TABLE DROP COLUMNXsafe operation, because your business logic shouldn't operate with this column on migration time, however betterALTER TABLE ALTER COLUMN DROP NOT NULL,ALTER TABLE DROP CONSTRAINTandDROP INDEXbefore * and *****18ALTER TABLE RENAME COLUMNadd new column and copy dataunsafe operation, it's too hard write business logic that operate with two columns simultaneously, so proposeALTER TABLE CREATE COLUMNand then copy all data to new column *19ALTER TABLE ADD CONSTRAINT CHECKadd as not valid and validateunsafe operation, because you spend time in migration to check constraint20ALTER TABLE DROP CONSTRAINT(CHECK)Xsafe operation21ALTER TABLE ADD CONSTRAINT FOREIGN KEYadd as not valid and validateunsafe operation, because you spend time in migration to check constraint, lock two tables22ALTER TABLE DROP CONSTRAINT(FOREIGN KEY)Xsafe operation, lock two tables23ALTER TABLE ADD CONSTRAINT PRIMARY KEYadd index and add constraintunsafe operation, because you spend time in migration to create index ***24ALTER TABLE DROP CONSTRAINT(PRIMARY KEY)Xsafe operation ***25ALTER TABLE ADD CONSTRAINT UNIQUEadd index and add constraintunsafe operation, because you spend time in migration to create index ***26ALTER TABLE DROP CONSTRAINT(UNIQUE)Xsafe operation ***27ALTER TABLE ADD CONSTRAINT EXCLUDEadd new table and copy data28ALTER TABLE DROP CONSTRAINT (EXCLUDE)X29CREATE INDEXCREATE INDEX CONCURRENTLYunsafe operation, because you spend time in migration to create index30DROP INDEXXDROP INDEX CONCURRENTLYsafe operation ***31CREATE INDEX CONCURRENTLYXsafe operation32DROP INDEX CONCURRENTLYXsafe operation ****: main point with migration on production without downtime that your code should correctly work before and after migration, lets look this point closely inDealing with logic that should work before and after migrationsection.**: postgres will check that all items in columnNOT NULLthat take time, lets look this point closely inDealing withNOT NULLconstraintsection.***: postgres will have same behaviour when you skipALTER TABLE ADD CONSTRAINT UNIQUE USING INDEXand still unclear difference withCONCURRENTLYexcept difference in locks, lets look this point closely inDealing withUNIQUEconstraint.****: lets look this point closely inDealing withALTER TABLE ALTER COLUMN TYPEsection.*****: if you check migration on CI withpython manage.py makemigrations --checkyou can't drop column in code without migration creation, so in this case you can be usefulback migration flow: apply code on all instances and then migrate databaseDealing with logic that should work before and after migrationAdding and removing models and columnsMigrations:CREATE SEQUENCE,DROP SEQUENCE,CREATE TABLE,DROP TABLE,ALTER TABLE ADD COLUMN,ALTER TABLE DROP COLUMN.This migrations are pretty safe, because your logic doesn't work with this data before migrationChanges for working logicMigrations:ALTER TABLE RENAME TO,ALTER TABLE SET TABLESPACE,ALTER TABLE RENAME COLUMN,ALTER TABLE ADD CONSTRAINT EXCLUDE.For this migration too hard implement logic that will work correctly for all instances, so there are two ways to dealing with it:create new table/column, copy exist data, drop old table/columndowntimeCreate column with defaultMigrations:ALTER TABLE ADD COLUMN SET DEFAULT.Standard django's behaviour for creation column with default is populate all values with default. Django don't use database defaults permanently, so when you add new column with default django will create column with default and drop this default at once, eg. new default will come from django code. In this case you can have a gap when migration applied by not all instances has updated and at this moment new rows in table will be without default and probably you need update nullable values after that. So to avoid this case best way is avoid creation column with default and split column creation (with default for new rows) and data population to two migrations (with deployments).Dealing withNOT NULLconstraintPostgres check that all column itemsNOT NULLwhen you applyingNOT NULLconstraint, for postgres 12 and newest it doesn't make this check if appropriateCHECK CONSTRAINTexists, but for older versions you can't defer this check as forNOT VALID. Fortunately we have some hacks and alternatives there for old postgres versions.Run migrations when load minimal to avoid negative affect of locking.SET statement_timeoutand try to setNOT NULLconstraint for small tables.UseCHECK (column IS NOT NULL)constraint instead that supportNOT VALIDoption with nextVALIDATE CONSTRAINT, see article for detailshttps://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c. There are additionally can be appliedNOT NULLconstraint via directpg_catalog.pg_attributeattnotnullupdate, but it require superuser permissions.Dealing withUNIQUEconstraintPostgres has two approaches for uniqueness:CREATE UNIQUE INDEXandALTER TABLE ADD CONSTRAINT UNIQUE- both use unique index inside. Difference that we can find that we cannot applyDROP INDEX CONCURRENTLYfor constraint. However it still unclear what difference forDROP INDEXandDROP INDEX CONCURRENTLYexcept difference in locks, but as we seen before both marked as safe - we don't spend time inDROP INDEX, just wait for lock. So as django use constraint for uniqueness we also have a hacks to use constraint safely.Dealing withALTER TABLE ALTER COLUMN TYPENext operations are safe:varchar(LESS)tovarchar(MORE)where LESS < MOREvarchar(ANY)totextnumeric(LESS, SAME)tonumeric(MORE, SAME)where LESS < MORE and SAME == SAMEFor other operations propose to create new column and copy data to it. Eg. some types can be also safe, but you should check yourself.django-pg-zero-downtime-migrations changelog0.14fixed deferred sql errorsadded django 5.0 supportadded python 3.12 supportadded postgres 16 supportdrop postgres 11 supportdropZERO_DOWNTIME_MIGRATIONS_USE_NOT_NULLsettingmarkedmigrate_isnotnull_check_constraintscommand deprecated0.13added django 4.2 supportmarked django 3.2 support deprecatedmarked django 4.0 support deprecatedmarked django 4.1 support deprecatedmarked postgres 11 support deprecateddrop postgres 10 supportupdated test docker image to ubuntu 22.040.12addedserialandinteger,bigserialandbigint,smallserialandsmallint, same types changes as safe migrationsfixedAutoFieldtype changing and concurrent insertions issue fordjango<4.1added sequence dropping and creation timeouts as they can be used withCASCADEkeyword and affect other tablesadded django 4.1 supportadded python 3.11 supportadded postgres 15 supportmarked postgres 10 support deprecateddrop django 2.2 supportdrop django 3.0 supportdrop django 3.1 supportdrop postgres 9.5 supportdrop postgres 9.6 supportadd github actions checks for pull requests0.11fixed rename model with keepingdb_tableraisesALTER_TABLE_RENAMEerror #26added django 3.2 supportadded django 4.0 supportadded python 3.9 supportadded python 3.10 supportadded postgres 14 supportmarked django 2.2 support deprecatedmarked django 3.0 support deprecatedmarked django 3.1 support deprecatedmarked python 3.6 support deprecatedmarked python 3.7 support deprecatedmarked postgres 9.5 support deprecatedmarked postgres 9.6 support deprecatedmove to github actions for testing0.10added django 3.1 supportadded postgres 13 supportdrop python 3.5 supportupdated test environment0.9fixed decimal to float migration errorfixed django 3.0.2+ tests0.8added django 3.0 supportadded concurrently index creation and removal operationsadded exclude constraint support as unsafe operationdrop postgres 9.4 supportdrop django 2.0 supportdrop django 2.1 supportdrop deprecateddjango_zero_downtime_migrations_postgres_backendmodule0.7added python 3.8 supportadded postgres specific indexes supportimproved tests clearnessfixed regexp escaping warning for management commandfixed style checkimproved READMEmarked python 3.5 support deprecatedmarked postgres 9.4 support deprecatedmarked django 2.0 support deprecatedmarked django 2.1 support deprecated0.6markedZERO_DOWNTIME_MIGRATIONS_USE_NOT_NULLoption deprecated for postgres 12+added management command for migration to realNOT NULLfromCHECK IS NOT NULLconstraintadded integration tests for pg 12, pg 11 root, pg 11 compatible not null constraint, pg 11 standard not null constraint and pg 10, 9.6, 9.5, 9.4, postgis databasesfixed compatible check not null constraint deletion and creation via pg_attribute bugsminimized side affect with deferred sql execution between operations in one migration moduleadded postgres 12 safeNOT NULLconstraint creationadded safeNOT NULLconstraint creation for extra permissions forpg_catalog.pg_attributewithZERO_DOWNTIME_MIGRATIONS_USE_NOT_NULL=USE_PG_ATTRIBUTE_UPDATE_FOR_SUPERUSERoption enabledmarkedAddFieldwithnull=Falseparameter and compatibleCHECK IS NOT NULLconstraint option as unsafe operation and avoidZERO_DOWNTIME_MIGRATIONS_USE_NOT_NULLvalue in this caseadded version to packagefixed pypi README images linksimproved README0.5extracted zero-downtime-schema to mixin to allow use this logic with other backendsmoved module fromdjango_zero_downtime_migrations_postgres_backendtodjango_zero_downtime_migrations.backends.postgresmarkeddjango_zero_downtime_migrations_postgres_backendmodule as deprecatedadded postgis backend supportimproved README0.4changed defaults forZERO_DOWNTIME_MIGRATIONS_LOCK_TIMEOUTandZERO_DOWNTIME_MIGRATIONS_STATEMENT_TIMEOUTfrom0mstoNoneto get same with default django behavior that respect default postgres timeoutsadded updates to documentations with options defaultsadded updates to documentations with best options usagefixed adding nullable field with default had no error and warning issueadded links to documentation with issue describing and safe alternatives usage for errors and warningsadded updates to documentations with type casting workarounds0.3added django 2.2 support withMeta.indexesandMeta.constraintsattributesfixed python deprecation warnings for regexpremoved unusedTimeoutExceptionimproved README and PYPI description0.2added option that allow disablestatement_timeoutfor long operations like index creation on constraint validation when statement_timeout set globally0.1.1added long description content type0.1replaced default sql queries with more safeadded options forstatement_timeoutandlock_timeoutadded option forNOT NULLconstraint behaviouradded option for unsafe operation restriction