Package: django-mass-migration
Django Mass MigrationA Django app which provides utilities for performing data operations on (optionally) large datasets.Similar to Django's built-in migration system, it allows you to define data migrations to be performed on the database,
and to apply those migration operations to your database, and it tracks which migrations have been applied and which haven't.Unlike Django's built-in migration system, it is designed for performing long-running data modification tasks
where there could be a significant amount of time between when the migration is started and when it's finished.
SeeConcepts.
It is particularly useful for schemaless databases where Django's normal concept of schema changes doesn't apply.Due to the expectation that migrations maybe be long-running, and just for awesomeness, it provides a web based (rather than terminal based) interface for managing the migrations.The actual running of the operations can be done on a "backend" of your choosing, e.g. a task queue.
A backend for running migration operations onDjangaewithGlcoudcis bundled with the app.InstallationInstall the package:pip install django-mass-migrationAddmassmigrationtosettings.INSTALLED_APPS.Addpath("migrations/", include("massmigration.urls"))to your root urlconf.If you're using the bundled backend, setsettings.MASSMIGRATION_TASK_QUEUEto a Google Cloud Tasks queue name.Creating A MigrationTo create a new migration, run themakemassmigrationmanagement command:python manage.py makemassmigration myapp my_migration_nameThis command has two required, positional arguments:app_label- this should be the name of an installed app in your project.migration_name- this should be the name that you want to give you migration.The command also takes the following optional arguments:--template- this should be the name of one of the supplied templates inmassmigration/templates/migration_templates, without the extension, e.g.--template=mapper. SeeMigration Types. The default ismapper.A file will be created inside a folder called 'massmigrations' in your app, using the supplied migration name.
E.g.myapp/massmigrations/001_my_migration.py.
This is a blank slate for you to write your migration code in.Migration TypesThere are three broad types of migration for you to choose from.simpleThis is for a migration which is just a single Python function.
It's intended for small operations which you know your backend can perform in one step.mapperThis is for mapping a function over the instances of a Django queryset.
You define a queryset and anoperationfunction, and the migration will call that function on each object in the queryset.The backend must be able to handle iterating over the queryset.
The bundled DjangaeBackend can handle almost infinite sized querysets.customIf you want to take matters into your own hands you can write an entirely custom migration.
These can still be tracked the same as the other operations, but the implementation of what your migration
does and how your the backend handles it are up to you.Applying MigrationsThere are two ways to apply a migration to the database:Via the Web UIOptionally: ensure that the Django admin is set up in your project.Deploy your project, including the new migration file.Either:Go to the Django Admin site and under Mass Migrations -> Migration Record, click the "Manage Migrations" button; orGo directly to the URL ofreverse("massmigration_manage")whatever path you've configured that to be on.Next to your new migration, click "Run...".Click "Run migration".Wait for the migration to be listed as applied in the Migration Record list view, or check the logging from your backend.ProgrammaticallyIf you want to create your own system for applying migrations, you can use the API functions.
Or you will be able to, once I've written them.Protecting Code Which Requires MigrationsSometimes in the development cycle of your application, you will want to ensure that a particular migration has been run,
usually when you've added new code which expects the migration to have been applied before the code is run.To handle this, Mass Migration provides the following utilities:massmigration.enforcement.requires_migrationThis is a function decorator which ensures that the function cannot be executed until the specified migration has been applied.
For example:@requires_migration(("myapp","0001_my_migration"))defmy_function():...This will raisemassmigration.exceptions.RequiredMigrationNotAppliedif the function is called before the migration is applied.Notes:If the migrationisapplied, then this will cache that fact, so it will only query the database the first time the function is called.The migration identifier can either be a tuple of(app_label, migration_name)or can be a string of"app_label:migration_name".There is an optional second argumentskip_in_tests, which defaults toTrue.massmigration.enforcement.view_requires_migrationThis is the same asrequires_migrationbut for decorating a view function.
If the specified migration is not applied then the view will return a 503 response.ConceptsGeneral approachTODO: long-running migrations, errors, etc.
Why long-running migrations are not run as part of the deployment process (because they take too long and you don't want to take your site down while they run).Model StateTODO: Stuff about why there's no model history (apps/schema_editor).Workflow & Code ProtectionTODO: Deployment workflow and use of enforcement utilitiesSettingsThe following settings can be used:MASSMIGRATION_BACKENDThis should be a dotted path string to the backend class that you want to use.
The default is"massmigration.backends.djangae.DjangaeBackend",
which works with SQL, Google Cloud Datastore and Firestore databases.MASSMIGRATION_TASK_QUEUEUsed by theDjangaeBackend, this sets the Google Cloud Tasks queue name to be used for running migration tasks.MASSMIGRATION_RECORD_CACHE_TIMEOUTYou're unlikely to need this.
It sets the time for caching MigrationRecords for the purpose of checking a migration's status during mapper operations.