Package: django-sequences
By default, Django gives each model an auto-incrementing integer primary key.
These primary keys look like they generate a continuous sequence of integers.However, this behavior isn’t guaranteed.If a transaction inserts a row and then is rolled back, the sequence counter
isn’t rolled back for performance reasons, creating a gap in primary keys.Such gaps may happen on all databases natively supported by Django:PostgreSQLMariaDB/ MySQLOracleSQLiteThey may also happen on most databases supported via third-party backends.This can cause compliance issues for some use cases such as accounting.This risk isn’t well known. Since most transactions succeed, values look
sequential. Gaps will only be revealed by audits.django-sequences solves this problem with aget_next_valuefunction
designed to be used as follows:fromdjango.dbimporttransactionfromsequencesimportget_next_valuefrominvoices.modelsimportInvoicewithtransaction.atomic():Invoice.objects.create(number=get_next_value("invoice_numbers"))Or, if you’d rather use an object-oriented API:fromdjango.dbimporttransactionfromsequencesimportSequencefrominvoices.modelsimportInvoiceinvoice_numbers=Sequence("invoice_numbers")withtransaction.atomic():Invoice.objects.create(number=next(invoice_numbers))get_next_valuerelies on the database’s transactional integrity to ensure
that each value is returned exactly once. As a consequence,the guarantees
of django-sequences apply only if you callget_next_valueand save its
return value to the database within the same transaction!Table of contentsGetting startedAPIDatabase supportMultiple databasesIsolation levelsContributingReleasingChangelogGetting starteddjango-sequences is tested with Django Django 3.2 (LTS), 4.0, 4.1, 4.2 (LTS),
and 5.0.It is also tested with all database backends built-in to Django: MySQL/MariaDB,
Oracle, PostgreSQL and SQLite.It is released under the BSD license, like Django itself.Install django-sequences:$pipinstalldjango-sequencesAdd it to the list of applications in your project’s settings:INSTALLED_APPS=[...,"sequences.apps.SequencesConfig",...]Run migrations:$django-adminmigrateAPIget_next_value>>>fromsequencesimportget_next_valueThis function generates a gapless sequence of integer values:>>>get_next_value()1>>>get_next_value()2>>>get_next_value()3It supports multiple independent sequences:>>>get_next_value("cases")1>>>get_next_value("cases")2>>>get_next_value("invoices")1>>>get_next_value("invoices")2The first value defaults to 1. It can be customized:>>>get_next_value("customers",initial_value=1000)# pro growth hackingTheinitial_valueparameter only matters whenget_next_valueis called
for the first time for a given sequence — assuming the corresponding database
transaction gets committed; as discussed above, if the transaction is rolled
back, the generated value isn’t consumed. It’s also possible to initialize a
sequence in a data migration and not useinitial_valuein actual code.Sequences can loop:>>>get_next_value("seconds",initial_value=0,reset_value=60)When the sequence reachesreset_value, it restarts atinitial_value.
In other words, it generatesreset_value - 2,reset_value - 1,initial_value,initial_value + 1, etc. In that case, each call toget_next_valuemust provideinitial_valuewhen it isn’t the default
andreset_value.Database transactions that callget_next_valuefor a given sequence
are serialized.As a consequence, when you callget_next_valuein a
database transaction, other callers trying to get a value from the same
sequence block until the transaction completes, either with a commit or a
rollback. You should keep such transactions short to minimize the impact on
performance.This is why databases default to a faster behavior that may create gaps.Passingnowait=Truemakesget_next_valueraise an exception instead of
blocking in this scenario. This is rarely useful. Also it doesn’t work for the
first call. (This is a bug but it’s harmless and hard to fix.)Calls toget_next_valuefor distinct sequences don’t interact with one
another.Finally, passingusing="..."allows selecting the database on which the
current sequence value is stored. When this parameter isn’t provided, it
defaults to the default database for writing models of thesequencesapplication. SeeMultiple databasesfor details.To sum up, the complete signature ofget_next_valueis:get_next_value(sequence_name="default",initial_value=1,reset_value=None,*,nowait=False,using=None,)get_next_values>>>fromsequencesimportget_next_valuesThis function generates values in batch:>>>get_next_values(10)range(1, 11)>>>get_next_values(10)range(11, 21)As a reminder, you must save all these values to the database within the same
transaction to benefit from the guarantees of django-sequences.get_next_valuessupports the same arguments asget_next_valueexceptreset_value.The complete signature ofget_next_valuesis:get_next_values(batch_size,sequence_name="default",initial_value=1,*,nowait=False,using=None,)get_last_value>>>fromsequencesimportget_last_valueThis function returns the last value generated by a sequence:>>>get_last_value()None>>>get_next_value()1>>>get_last_value()1>>>get_next_value()2>>>get_last_value()2If the sequence hasn’t generated a value yet,get_last_valuereturnsNone.It supports independent sequences likeget_next_value:>>>get_next_value("cases")1>>>get_last_value("cases")1>>>get_next_value("invoices")1>>>get_last_value("invoices")1It acceptsusing="..."for selecting the database on which the current
sequence value is stored, defaulting to the default database for reading
models of thesequencesapplication.The complete signature ofget_last_valueis:get_last_value(sequence_name="default",*,using=None,)get_last_valueis a convenient and fast way to tell how many values a
sequence generated but it makes no guarantees.Concurrent calls toget_next_valuemay produce unexpected results ofget_last_value.delete>>>fromsequencesimportdeleteThis function deletes a sequence. It returnsTrueif the sequence existed
and was deleted andFalseotherwise.>>>company_id="b1f6cdef-367f-49e4-9cf5-bb0d34707af8">>>get_next_value(f"invoices—{company_id}")1>>>delete(f"invoices—{company_id}")True>>>delete(f"invoices—{company_id}")FalseIt acceptsusing="..."for selecting the database, likeget_next_value.The complete signature ofdeleteis:delete(sequence_name="default",*,using=None,)deleteis useful when you create many sequences and want to dispose of some.Sequence>>>fromsequencesimportSequence(not to be confused withsequences.models.Sequence, a private API)This class stores parameters for a sequence and providesget_next_value,get_next_values,get_last_value, anddeletemethods:>>>claim_ids=Sequence("claims")>>>claim_ids.get_next_value()1>>>claim_ids.get_next_value()2>>>claim_ids.get_last_value()2>>>claim_ids.delete()TrueThis reduces the risk of errors when the same sequence is used in multiple
places.Instances ofSequenceare also infinite iterators:>>>next(claim_ids)3>>>next(claim_ids)4The complete API is:Sequence(sequence_name="default",initial_value=1,reset_value=None,*,using=None,)Sequence.get_next_value(self,*,nowait=False,)Sequence.get_next_values(self,batch_size,*,nowait=False,)Sequence.get_last_value(self,)Sequence.delete(self,)All parameters have the same meaning as in theget_next_value,get_last_values,get_last_value, anddeletefunctions.ExamplesPer-date sequencesIf you want independent sequences per day, month, or year, use the appropriate
date fragment in the sequence name. For example:fromdjango.utilsimporttimezonefromsequencesimportget_next_value# Per-day sequenceget_next_value(f"books-{timezone.now().date().isoformat()}")# Per-year sequenceget_next_value(f"prototocol-{timezone.now().year}")The above calls will result in separate sequences likebooks-2023-03-15orprotocol-2024, respectively.Database supportdjango-sequences is tested on PostgreSQL, MariaDB / MySQL, Oracle, and SQLite.MySQL only supports thenowaitparameter from version 8.0.1.
MariaDB only supportsnowaitfrom version 10.3.Multiple databasesSince django-sequences relies on the database to guarantee transactional
integrity, the current value for a given sequence must be stored in the same
database as models containing generated values.In a project that uses multiple databases, you must write a suitable database
router to create tables for thesequencesapplication on all databases
storing models containing sequential numbers.Each database has its own namespace: a sequence with the same name stored in
two databases will have independent counters in each database.Isolation levelsSince django-sequences relies on the database’s transactional integrity, using
a non-default transaction isolation level requires special care.read uncommitted:django-sequences cannot work at this isolation level.Indeed, concurrent transactions can create gaps, as in this scenario:Transaction A reads N and writes N + 1;Transaction B reads N + 1 (dirty read) and writes N + 2;Transaction A is rolled back;Transaction B is committed;N + 1 is a gap.The read uncommitted isolation level doesn’t provide sufficient guarantees.
It will never be supported.read committed:django-sequences works best at this isolation level,
like Django itself.repeatable read:django-sequences also works at this isolation level,
provided your code handles serialization failures and retries transactions.This requirement isn’t specific to django-sequences. It’s generally needed
when running at the repeatable read isolation level.Here’s a scenario where only one of two concurrent transactions can
complete on PostgreSQL:Transaction A reads N and writes N + 1;Transaction B attemps to read; it must wait until transaction A completes;Transaction A is committed;Transaction B is aborted.On PostgreSQL, serialization failures are reported as:OperationalError: could not serialize access due to concurrent update.On MySQL, they result in:OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction').Concurrent transactions initializing the same sequence are also vulnerable,
although that’s hardly ever a problem in practice.On PostgreSQL, this manifests asIntegrityError: duplicate key value violates unique constraint "sequences_sequence_pkey".serializable:the situation is identical to the repeatable read level.SQLite always runs at the serializable isolation level. Serialization
failures result in:OperationalError: database is locked.ContributingPrepare a development environment:InstallPoetry.Runpoetry install.Runpoetry shellto load the development environment.Prepare testing databases:Install PostgreSQL, MariaDB, and Oracle.Create a database calledsequences, owned by a user calledsequenceswith passwordsequences, with permissions to create atest_sequencestest database. You may override these values with environment variables; seetests/*_settings.pyfor details.Make changes:Make changes to the code, tests, or docs.Runmake styleand fix errors.Runmake testto run the set suite on all databases.Iterate until you’re happy.Check quality and submit your changes:Installtox.Runtoxto test on all Python and Django versions and all databases.Submit a pull request.ReleasingIncrement version number X.Y inpyproject.toml.Commit, tag, and push the change:$gitcommit-m"Bump version number".$gittagX.Y$gitpush$gitpush--tagsBuild and publish the new version:$poetrybuild$poetrypublishChangelog3.0Add theget_next_valuesfunction for generating values in batch.2.9Add thedeletefunction.2.8No significant changes.2.7Sequence values can go up to2 ** 63 - 1instead of2 ** 31 - 1previously. The exact limit depends on the database backend.Migration0002_alter_sequence_last.pychanges the field storing sequence
values fromPositiveIntegerFieldtoPositiveBigIntegerField. Running
it requires an exclusive lock on the table, which prevents other operations,
including reads.If you have many distinct sequences, e.g. if you create one sequence per user
and you have millions of users, review how the migration will affect your app
before running it or skip it withmigrate--fake.2.6Improve documentation.2.5Fix Japanese and Turkish translations.Restore compatibility with Python 3.5.Support relabeling thesequencesapp with a customAppConfig.2.4Add theget_last_valuefunction.Add theSequenceclass.2.3Optimize performance on MySQL.Test on MySQL, SQLite and Oracle.2.2Optimize performance on PostgreSQL ≥ 9.5.2.1Provide looping sequences withreset_value.2.0Add support for multiple databases.Add translations.nowaitbecomes keyword-only argument.Drop support for Python 2.1.0Initial stable release.