Package: migrate-anything
Migrate anything - database (etc.) migration utility, especially for Python projects.What is this?It’s kinda annoying how often you run into the question of how to handle migrations in your project, and there hasn’t seem to emerged any good, DB -agnostic, framework-agnostic, and storage-agnostic tool to manage them.This project is an attempt to change that.Basically what it does when you runmigrate-anything migrationsis:Find all the filesmigrations/*.pyand sort themAny that are not yet registered in the DB will be loaded, theirup()is executed, and the file’s contents stored in the DBAny files that are missing from the fs but are in the DB will have their code loaded from the DB and theirdown()is executed - in reverse orderLicenseLicensing is important. This project uses BSD 3-clause license, and adds no other dependencies to your project (it does use a few things during build & testing) - that’s about as simple, safe, and free to use as it gets.For more information check theLICENSE-file.Usage examplesBasic usageFirstly you’ll need this package in your project. Pick one of these:pipinstall-Umigrate-anythingpoetryaddmigrate-anythingpipenvinstallmigrate-anythingSimply put, create a Python package, don’t be too clever and call it e.g.migrations. Then put files in that package:# migrations/__init__.pyfrommigrate_anythingimportconfigure,CSVStorageconfigure(storage=CSVStorage("migration_status.csv"))# migrations/01-initialize-db.py# Please note that this is built for a completely hypothetical DB layerfrommy_dbimportget_dbDB=get_db()defup():DB.create_table("example")defdown():DB.delete_table("example")This would configure your migrations’ status to be stored in a local file calledmigration_status.csvand set up your first migration script. If you have amy_dbmodule that works like this you could just run this with a single command:migrate-anythingmigrationspoetryrunmigrate-anythingmigrationspipenvrunmigrate-anythingmigrationsNow in the real world you might want something more durable and a realistic example, so here’s e.g. what you’d do when using MongoDB:# __init__.pyfrommigrate_anythingimportconfigure,MongoDBStorageimportpymongodb=pymongo.MongoClient().my_dbconfigure(storage=MongoDBStorage(db.migrations))# 01-initialize-db.pyfrompymongoimportMongoClientclient=MongoClient()db=client.my_dbdefup():db.posts.insert_one({"id":"post-1","title":"We're live!","content":"This is our first post, yay."})db.posts.create_index("id")defdown():db.posts.drop()This would configure storage to amy_db.migrationsMongoDB collection.Command line flags# Revert the last migration using migration code file.migrate-anythingmigrations--revert-latestCustom storage enginesWriting your own custom storage engine is easy.# __init__.pyfrommigrate_anythingimportconfigureclassCustomStorage(object):def__init__(self,file):self.file=filedefsave_migration(self,name,code):withopen(self.file,"a",encoding="utf-8")asfile:file.write("{},{}\n".format(name,code))deflist_migrations(self):try:withopen(self.file,encoding="utf-8")asfile:return[line.split(",")forlineinfile.readlines()ifline.strip()# Skip empty lines]exceptFileNotFoundError:return[]defremove_migration(self,name):migrations=[migrationformigrationinself.list_migrations()ifmigration[0]!=name]withopen(self.file,"w",encoding="utf-8")asfile:forrowinmigrations:file.write("{},{}\n".format(*row))configure(storage=CustomStorage("test.txt"))You can also check out theexamples.ContributingThis project is run on GitHub using the issue tracking and pull requests here. If you want to contribute, feel free tosubmit issues(incl. feature requests) or PRs here.You will needpre-commitset up to make contributions.To set up development tools for this, run:pre-commitinstallvirtualenv.venv.venv/bin/activate# OR.venv\Scripts\activate.batpipinstall-rdev-requirements.txtpipinstall-e.And then to run the testspytestWhen you have improvements to make, commit (and include any cleanup pre-commit might do), push your changes to your own fork, make a PR.Future ideasFuture ideas include support for other DB engines (feel free to contribute),
and Kubernetes ConfigMap. Annoyingly storage to Kubernetes from inside a pod
and in code is not quite as simple as just runningkubectl.Oh and your Kubernetes pods will likely require the necessary RBAC rules to manage their ConfigMap. It’s unfortunately kinda complex, but I’m sure you can figure it out e.g. with thisguide.Financial supportThis project has been made possible thanks toCocreatorsandLietu. You can help us continue our open source work by supporting us onBuy me a coffee.