Package: metapensiero.sphinx.patchdb
Version:2.0Author:Lele Gaifax <lele@metapensiero.it>license:GPLv3Building and maintaining the schema of a database is always a challenge. It may quickly become
a nightmare when dealing with even moderately complex databases, in a distribuited development
environment. You have new features going in, and fixes here and there, that keeps accumulating
in the developmentbranch. You also have several already deployed instances of the database
you wanna upgrade now and then.In my experience, it’s very difficult to impossible to come up with a completely automated
solution, for several reasons:comparison between different releases of a database schema is trickyactual contents of the database must be preservedsome changes require specific recipes to upgrade the dataany automated solution hides some detail, by definition: I need complete control, to be able
to create temporary tables and/or procedures for exampleI tried, and wrote myself, several different approaches to the problem[*], and this package
is my latest and most satisfying effort: it builds on top ofdocutilsandSphinx, with
the side advantage that you get a quite nice and good documentation of the whole architecture:literate database scheming!ContentsHow it worksScriptsConditionsVariablesDependenciesErrors handlingPatchesObsoleted patchesRun-always scriptsFake data domainsPlaceholdersUsageCollecting patchesUpdating the databaseAutomatic backupSupported databasesExample development Makefile snippetQuick exampleChanges3.7 (2019-12-20)3.6 (2019-12-19)3.5 (2019-06-21)3.4 (2019-03-31)3.3 (2019-03-31)3.2 (2018-03-03)3.1 (2017-11-30)3.0 (2017-11-06)[*]Just to mention a few alternatives:AlembicWritten on top ofSQLAlchemyby the same author: it does not help when you need to
manage something outside SA knowledge (stored procedures, permissions, …)SqlibristSome similarities with PatchDB, very young, Django integration.SqitchQuite good,althoughPerl based…See theschema migrationpage on
Wikipedia for further details.How it worksThe package contains two distinct pieces: aSphinxextension and thepatchdbcommand
line tool.The extension implements a newReSTdirective able to embed ascriptin the document: when
processed by thesphinx-buildtool, all the scripts will be collected in an external file,
configurable.Thepatchdbtool takes that script collection and determines which scripts need to be
applied to some database, and the right order.It creates and maintains a single very simple table within the database (unsurprisingly namedpatchdb), where it records the last version of each script it successfully execute, so that
it won’t reexecute the same script (actually, a particularrevisionof it) twice.So, on the development side you simply write (and document!) each piece, and when it comes the
time of deploying current state you distribute just the script collection (a single file,
usually inAXON,JSONorYAMLformat, or apicklearchive, seestorage formatsbelow) to the end points where the database instances live, and executepatchdbagainst
each one.ScriptsThe basic building block is ascript, an arbitrary sequence of statements written in some
language (currently, eitherPython,SQLorShell), augmented with some metadata
such as thescriptid, possibly a longerdescription, itsrevisionand so on.As a complete example of the syntax, consider the following:.. patchdb:script:: My first script
   :description: Full example of a script
   :revision: 2
   :depends: Other script@4
   :preceeds: Yet another
   :language: python
   :conditions: python_2_x

   print "Yeah!"This will introduce a script globally identified byMy first script, written inPython:
this is its second release, and its execution must be constrained such that it happensafterthe execution of the fourth revision ofOther scriptandbeforeYet another.The sequence of statements may be specified either as thecontentof the directiveorloaded from an external file, so the previous script could be written as:.. patchdb:script:: My first script
   :description: Full example of a script
   :revision: 2
   :depends: Other script@4
   :preceeds: Yet another
   :language: python
   :conditions: python_2_x
   :file: python_script.pySQLscripts may be composed by multiple statements, separated by a standalone;;marker, as in:.. patchdb:script:: Create and populate

   CREATE TABLE foo (id integer, value varchar(20))
   ;;
   INSERT INTO foo (id, value) VALUES (1, 'bar')Another special marker is;;INCLUDE:, that can be used to include the content of an
external file, more flexibly than with thefileoption above. The previous example could be
written as:.. patchdb:script:: Create and populate

   ;;INCLUDE: create_table.sql
   ;;
   ;;INCLUDE: populate_table.sqlwhere the two statements are loaded respectively fromcreate_table.sqlandpopulate_table.sql. The;;INCLUDE:marker is expanded recursively, so that another way
to say the very same thing is:.. patchdb:script:: Create and populate

   ;;INCLUDE: create_and_populate.sqlwithcreate_and_populate.sqlcontaining:;;INCLUDE: create_table.sql
;;
;;INCLUDE: populate_table.sqlAs another concrete example where this can be very useful, consider the case when you need to
replace an existing function with one having a different signature for the output parameters,
something that for example PostgreSQL does not allow. You could then say:.. patchdb:script:: Some function
   :revision: 2
   :file: some_function.sql

.. patchdb:script:: Upgrade some function to revision 2
   :depends: Some function@1
   :brings: Some function@2

   DROP FUNCTION some_function(int, OUT int)
   ;;
   ;;INCLUDE: some_function.sqlConditionsThe example shows also an usage of the conditions, allowing more than one variant of a script
like:.. patchdb:script:: My first script (py3)
   :description: Full example of a script
   :revision: 2
   :depends: Other script@4
   :preceeds: Yet another
   :language: python
   :conditions: python_3_x

   print("Yeah!")The value of the:conditions:option may be a single paragraph, containing a comma
separated list of conditions, or alternatively abullet list.As another use case of this feature, the following snippet declares the same table for two
different databases:.. patchdb:script:: Simple table (PostgreSQL)
  :language: sql
  :mimetype: text/x-postgresql
  :conditions: postgres
  :file: postgresql/simple.sql

.. patchdb:script:: Simple table (MySQL)
  :language: sql
  :mimetype: text/x-mysql
  :conditions: mysql
  :file: mysql/simple.sqlAs you can see, the content of the script can be conveniently stored in an external file, and
the particular dialect specified with the:mimetype:option, so it will be properly
highlighted by Pygments.Such conditions may also be arbitrarily defined on the command line, so you can have for
example:.. patchdb:script:: Configure for production
  :language: sql
  :conditions: PRODUCTION

  UPDATE configuration SET is_production = trueand then add the option--assertPRODUCTIONwhen it is the case.A condition can benegated, prepending a!to its name:.. patchdb:script:: Configure for production
  :language: sql
  :conditions: !PRODUCTION

  UPDATE configuration SET is_production = falseVariablesAnother way to influence a script effect is by usingvariables: a script may contain one or
more references to arbitrary variables using the syntax{{VARNAME}}, thatmustbe
defined at application time, using the--defineVARNAME=VALUEcommand line option.
Alternatively with the syntax{{name=default}}the reference can set the default value for
the variable, that can be overridden from the command line.As an example, you can have the following script:.. patchdb:script:: Create table and give read-only rights to the web user
   :language: sql

   CREATE TABLE foo (id INTEGER)
   ;;
   GRANT SELECT ON TABLE foo TO {{WEB=www}}
   ;;
   GRANT ALL ON TABLE foo TO {{ADMIN}}To apply it, you must specify the value for theADMINvariable, with something like--defineADMIN=$USER.The variable name must be anidentifier(that is, at least an alphabetic letter possibly
followed by alphanumerics or underscores), while its value may contain whitespaces, letters or
digits.If the name starts withENV_, the value is looked up in the processenvironment. In the
following example, the name of the user is taken from the theUSERenvironment variable
(that must be present), while the password comes from thePASSWORDenvironment entry or, if
not set, from the specified default:.. patchdb:script:: Insert a default user name
   :language: sql

   INSERT INTO users (name, password) VALUES ('{{ENV_USER}}', '{{ENV_PASSWORD=password}}')Note that you can override the environment using an explicit--defineoption on the command
line, for example with--defineENV_PASSWORD=foobar.DependenciesThe dependencies (that is, theoptions:brings:,:depends:,:drops::and:preceeds:) may be a paragraph containing a comma separated list of script ids, such as:.. patchdb:script:: Create master table

   CREATE TABLE some_table (id INTEGER PRIMARY KEY, tt_id INTEGER)

.. patchdb:script:: Create target table

   CREATE TABLE target_table (id INTEGER PRIMARY KEY)

.. patchdb:script:: Add foreign key to some_table
   :depends: Create master table, Create target table

   ALTER TABLE some_table
         ADD CONSTRAINT fk_master_target
             FOREIGN KEY (tt_id) REFERENCES target_table (id)WarningThis implies that the referencedscriptidscannotinclude a comma.Alternatively, they can be entered as abullet list, so the last script above can be written
also as:.. patchdb:script:: Add foreign key to some_table
   :depends:
      - Create master table
      - Create target table

   ALTER TABLE some_table
         ADD CONSTRAINT fk_master_target
             FOREIGN KEY (tt_id) REFERENCES target_table (id)With this syntax you can reference ascriptidcontaining a comma.Independently from the order these scripts appear in the documentation, the third script will
execute only after the first two are successfully applied to the database. As you can notice,
most of the options are optional: by default,:language:issql,:revision:is1, the:description:is taken from the title (that is, the script ID), while:depends:and:preceeds:are empty.Just for illustration purposes, the same effect could be achieved with:.. patchdb:script:: Create master table
   :preceeds: Add foreign key to some_table

   CREATE TABLE some_table (id INTEGER PRIMARY KEY, tt_id INTEGER)

.. patchdb:script:: Create target table

   CREATE TABLE target_table (id INTEGER PRIMARY KEY)

.. patchdb:script:: Add foreign key to some_table
   :depends: Create target table

   ALTER TABLE some_table
         ADD CONSTRAINT fk_master_target
             FOREIGN KEY (tt_id) REFERENCES target_table (id)Errors handlingBy defaultpatchdbstops when it fails to apply one script. Some time you may want to relax
that rule, for example when operating on a database that was created with other methods so you
cannot relay on the existence of a specific script to make the decision. In such cases, the
option:onerror:may be used:.. patchdb:script:: Remove obsoleted tables and functions
   :onerror: ignore

   DROP TABLE foo
   ;;
   DROP FUNCTION initialize_record_foo()When:onerror:is set toignore, each statement in the script is executed and if an error
occurs it is ignored andpatchdbproceeds with the next one. On good databases like
PostgreSQL and SQLite where even DDL statements are transactional, each statement is executed
in a nested subtransaction, so subsequent errors do not ruin the effect of correctly applied
previous statements.Another possible setting of this option isskip: in this case, whenever an error occurs the
effect of the whole script is undone and it is considered as applied. For example, assuming
that the old version ofSomeProcedureaccepted a single argument and the new one requires
two of them, you could do something like the following:.. patchdb:script:: Fix stored procedure signature
   :onerror: skip

   SELECT somecol FROM SomeProcedure(NULL, NULL)
   ;;
   ALTER PROCEDURE SomeProcedure(p_first INTEGER, p_second INTEGER)
   RETURNS (somecol INTEGER) AS
   BEGIN
     somecol = p_first * p_second;
     SUSPEND;
   ENDPatchesApatchis a particular flavour of script, one that specifies abringsor adropsdependency list. Imagine that theexample abovewas the first version of the database, and
that the current version looks like the following:.. patchdb:script:: Create master table
   :revision: 2

   CREATE TABLE some_table (
     id INTEGER PRIMARY KEY,
     description VARCHAR(80),
     tt_id INTEGER
   )that is,some_tablenow contains one more field,description.We need an upgrade path from the first revision of the table to the second:.. patchdb:script:: Add a description to the master table
   :depends: Create master table@1
   :brings: Create master table@2

   ALTER TABLE some_table ADD COLUMN description VARCHAR(80)Whenpatchdbexamines the database status, it will execute oneorthe other. If the
scriptCreate master tableisn’t executed yet (for example when operating on a new database),
it will take the former script (the one that creates the table from scratch).  Otherwise, if
the database “contains” revision 1 (and not higher than 1) of the script, it will execute the
latter, bumping up the revision number.Obsoleted patchesAnother peculiarity of this kind of scripts is that they may referencesnon existing scriptswithout producing warnings or errors.The rationale is that in the database evolution a given script may be removed, possibly
replaced by a different one by some succeeding patch. Consider the case when you once had a
table calledcustomers:.. patchdb:script:: Create table customers
   :revision: 2

   CREATE TABLE customers (
     id SERIAL PRIMARY KEY,
     name VARCHAR(80),
     street_address VARCHAR(80),
     city VARCHAR(80),
     telephone_number VARCHAR(80)
   )

.. patchdb:script:: Add telephone number to customers table
   :depends: Create table customers@1
   :brings: Create table customers@2

   ALTER TABLE customers ADD COLUMN telephone_number VARCHAR(80)and then the need for multiple addresses arose thus you decided to split it in two distinct
relations, apersonsand aperson_addresses:.. patchdb:script:: Create table persons

   CREATE TABLE persons (
     id SERIAL PRIMARY KEY,
     name VARCHAR(80)
   )

.. patchdb:script:: Create table person_addresses
   :depends: Create table persons

   CREATE TABLE person_addresses (
     id SERIAL PRIMARY KEY,
     person_id INTEGER REFERENCES persons (id),
     street_address VARCHAR(80),
     city VARCHAR(80),
     telephone_number VARCHAR(80)
   )

.. patchdb:script:: Migrate from customers to persons and person_addresses
   :depends:
      - Create table customers@2
      - Create table persons
      - Create table person_addresses
   :drops:
      - Create table customers
      - Add telephone number to customers table

   INSERT INTO persons (id, name) SELECT id, name FROM customers
   ;;
   INSERT INTO person_addresses (person_id, street_address, city, telephone_number)
     SELECT id, street_address, city, telephone_number
     FROM customers
   ;;
   DROP TABLE customersAt that point the script that introduced the originalcustomerstable disappeared from the
documentation, but you most probably want to keep the migration patch around for a while, at
least until you are sure all your production databases got upgraded.HintIn the HTML output, missing dependencies such as the"Create table customers"above are marked with astrikeclass. In order to actually get a visual effect
you need to define theCSSstyle for that, for example putting the following in
the Sphinx configuration:html_context = {
    'css_files': [
        '_static/theme_overrides.css',
    ],
}and something likespan.strike {
  text-decoration: line-through;
}in_static/theme_overrides.css.Run-always scriptsYet another variant of scripts, which get applied always,every timepatchdbis
executed.  This kind may be used to perform arbitrary operations, either at the start or at the
end of thepatchdbsession:.. patchdb:script:: Say hello
   :language: python
   :always: first

   print("Hello!")

.. patchdb:script:: Say goodbye
   :language: python
   :always: last

   print("Goodbye!")Fake data domainsAs a special case that uses this kind of script, the following example illustrate anapproximationof thedata domainswith MySQL, that lacks them:.. patchdb:script:: Define data domains (MySQL)
   :language: sql
   :mimetype: text/x-mysql
   :conditions: mysql
   :always: first

   CREATE DOMAIN bigint_t bigint
   ;;
   CREATE DOMAIN `Boolean_t` char(1)

.. patchdb:script:: Create some table (MySQL)
   :language: sql
   :mimetype: text/x-mysql
   :conditions: mysql
   :always: first

   CREATE TABLE `some_table` (
       `ID` bigint_t NOT NULL,
     , `FLAG` `Boolean_t`

     , PRIMARY KEY (`ID`)
   )WarningThis is just a dirty hack, based on relatively simple search and replace: don’t
take it seriously, use a better database if you really needdata domains!NoteThis works also with SQLite.PlaceholdersAnother feature is that the definition of the database, that is the collection of the scripts
that actually define its schema, may be splitted on multiple Sphinx environments: the use case
is when you have a complex application, composed by multiple modules, each of them requiring
its own set of DB objects.A script is considered aplaceholderwhen it has an empty body: it won’t be ever applied, but
instead its presence in the database will be asserted. In this way, one Sphinx environment
could contain the following script:.. patchdb:script:: Create table a

   CREATE TABLE a (
       id INTEGER NOT NULL PRIMARY KEY
     , value INTEGER
   )and another documentation set could extend that with:.. patchdb:script:: Create table a
   :description: Place holder

.. patchdb:script:: Create unique index on value
   :depends: Create table a

   CREATE UNIQUE INDEX on_value ON a (value)The second set can be appliedonlyafter the former one is.UsageCollecting patchesTo use it, first of all you must register the extension within the Sphinx environment, adding
the full name of the package to theextensionslist in the fileconf.py, for example:# Add any Sphinx extension module names here, as strings.
extensions = ['metapensiero.sphinx.patchdb']The other required bit of customization is the location of theon disk scripts storage,
i.e. the path of the file that will contain the information about every found script: this is
kept separated from the documentation itself because you will probably deploy it on production
servers just to update their database.Storage formatsIf the filename ends with.jsonit will contain aJSONformatted array, if it ends
with.yamlthe information will be dumped inYAML, if it ends with.axonthe
dump will be formatted usingAXON, otherwise it will be a Pythonpickle. I usually
preferAXON,JSONorYAML, because those formats are more VCs friendly and open
to human inspection. These days I tend to useAXONfor this kind of things as it is
slightly more readable and more VCs friendly thanJSON, whileYAMLis very slow.The location may be set in the sameconf.pyas above, like:# Location of the external storage
patchdb_storage = '…/dbname.json'Otherwise, you can set it using the-Doption of thesphinx-buildcommand, so that you
can easily share its definition with other rules in aMakefile. I usually put the following
snippet at the beginning of theMakefilecreated bysphinx-quickstart:TOPDIR ?= ..
STORAGE ?= $(TOPDIR)/database.json

SPHINXOPTS = -D patchdb_storage=$(STORAGE)At this point, executing the usualmake htmlwill update the scripts archive: that file
contains everything is needed to update the database either local or remote; in other words,
running Sphinx (or even having it installed) isnotrequired to update a database.Updating the databaseThe other side of the coin is managed by thepatchdbtool, that digests the scripts archive
and is able to determine which of the scripts are not already applied and eventually does that,
in the right order.When your database does already exist and you are just starting usingpatchdbyou may need
to force the initial state with the following command:patchdb --assume-already-applied --postgresql "dbname=test" database.jsonthat will just update thepatchdbtable registering current revision of all the missing
scripts, without executing them.You can inspect what will be done, that is obtain the list of not already applied patches, with
a command like:patchdb --dry-run --postgresql "dbname=test" database.jsonThedatabase.jsonarchive can be sent to the production machines (in some cases I put it in aproductionbranch of the repository and use the version control tool to update the remote
machines, in other I simply usedscporrsyncbased solutions). Another way is to
include it in some package and then use the syntaxsome.package:path/database.json.The scripts may even come from several different archives (seeplaceholdersabove):patchdb --postgresql "dbname=test" app.db.base:pdb.json app.db.auth:pdb.jsonAutomatic backupIn particular in development mode, I find it useful to have a simple way of going back to a
previous state and retry the upgrade, either to test different upgrade paths or to fix silly
typos in the new patches.Since version 2.3patchdbhas a new option,--backups-dir, that controls an automatic
backup facility: at each execution, before proceeding with applying missing patches,regardlesswhether there are any, by default it takes a backup of the current database and
keeps a simple index of these snapshots.The option defaults to the system-wide temporary directory (usually/tmpon POSIX systems):
if you you don’t need the automatic backup (a reasonable production system should have a
different approach to taking such snapshots), specifyNoneas argument to the option.With thepatchdb-statestool you obtain a list of the available snapshots, or restore any
previous one:$ patchdb-states list
[lun 18 apr 2016 08:24:48 CEST] bc5c5527ece6f11da529858d5ac735a8 <create first table@1>
[lun 18 apr 2016 10:27:11 CEST] 693fd245ad9e5f4de0e79549255fbd6e <update first table@1>

$ patchdb-states restore --sqlite /tmp/quicktest.sqlite 693fd245ad9e5f4de0e79549255fbd6e
[I] Creating patchdb table
[I] Restored SQLite database /tmp/quicktest.sqlite from /tmp/693fd245ad9e5f4de0e79549255fbd6e

$ patchdb-states clean -k 1
Removed /tmp/bc5c5527ece6f11da529858d5ac735a8
Kept most recent 1 snapshotSupported databasesAs of version 2,patchdbcan operate on the following databases:Firebird (requiresfdb)MySQL (requiresPyMySQLby default, see option--driverto select a different one)PostgreSQL (requirespsycopg2)SQLite (uses the standard librarysqlite3module)Example development Makefile snippetThe following is a snippet that I usually put in my outerMakefile:export TOPDIR := $(CURDIR)
DBHOST := localhost
DBPORT := 5432
DBNAME := dbname
DROPDB := dropdb --host=$(DBHOST) --port=$(DBPORT) --if-exists
CREATEDB := createdb --host=$(DBHOST) --port=$(DBPORT) --encoding=UTF8
STORAGE := $(TOPDIR)/$(DBNAME).json
DSN := host=$(DBHOST) port=$(DBPORT) dbname=$(DBNAME)
PUP := $(PATCHDB) --postgresql="$(DSN)" --log-file=$(DBNAME).log $(STORAGE)

# Build the Sphinx documentation
doc:
        $(MAKE) -C doc STORAGE=$(STORAGE) html

$(STORAGE): doc

# Show what is missing
missing-patches: $(STORAGE)
        $(PUP) --dry-run

# Upgrade the database to the latest revision
database: $(STORAGE)
        $(PUP)

# Remove current database and start from scratch
scratch-database:
        $(DROPDB) $(DBNAME)
        $(CREATEDB) $(DBNAME)
        $(MAKE) databaseQuick exampleThe following shell session illustrates the basics:python3-mvenvpatchdb-sessioncdpatchdb-sessionsourcebin/activatepipinstallmetapensiero.sphinx.patchdb[dev]yesn|sphinx-quickstart--projectPatchDB-Quick-Test\--authorJohnDoe\-v1--release1\--languageen\--masterindex--suffix.rst\--makefile--no-batchfile\pdb-qtcdpdb-qtecho"extensions = ['metapensiero.sphinx.patchdb']">>conf.pyecho"patchdb_storage = 'pdb-qt.json'">>conf.pyecho"
.. patchdb:script:: My first script
   :depends: Yet another
   :language: python

   print('world!')

.. patchdb:script:: Yet another
   :language: python

   print('Hello')
">>index.rstmakehtmlpatchdb--sqlite/tmp/pdb-qt.sqlite--dry-runpdb-qt.jsonAt the end you should get something like:Would apply script "yet another@1"
Would apply script "my first script@1"
100% (2 of 2) |########################################| Elapsed Time: 0:00:00 Time: 0:00:00Removing the--dry-run:$ patchdb --sqlite /tmp/pdb-qt.sqlite pdb-qt.json
Hello
world!

Done, applied 2 scripts
100% (2 of 2) |########################################| Elapsed Time: 0:00:00 Time: 0:00:00Once again:$ patchdb --sqlite /tmp/pdb-qt.sqlite pdb-qt.json
Done, applied 0 scriptsChanges[1]3.7 (2019-12-20)Catch dependency error when a patch brings a script to a revision higher than its current3.6 (2019-12-19)Now Python scripts receive a reference to the current patch manager, so they are able to
execute arbitrary scripts already in the storage3.5 (2019-06-21)Now it’s an hard error when a patch brings an unknown script: when it does, it’s either
obsoleted or there is a typo somewhere3.4 (2019-03-31)Nothing new, minor glitch in the release procedure3.3 (2019-03-31)Lift the constraint on sqlparse version, allow use of recently released 0.3.0.3.2 (2018-03-03)Usepython-rapidjsonif available3.1 (2017-11-30)Fix glitch in the logic that determine whether a patch script is still validUseenlightento show the progress bar: the--verboseoption is gone, now is the
default mode3.0 (2017-11-06)Python 3 only[2]New execution logic, hopefully fixing circular dependencies error in case of multiple non
trivial pending migrations[1]Previous changes arehere.[2]If you are still using Python 2, either stick with version 2.27, or fetchthis
commitfrom the repository.