Package: slice-db
Slice DBOverviewSliceDB is a tool for capturing and restoring a subset of a PostgreSQL database.
It also supports scrubbing sensive data.InstallPippip3installslice-dbDockerdockerpullrivethealth/slicedbUsageFor all commands and options, seeUsage.Basic exampleFirst, query a database to create a schema file.slicedbschema>schema.ymlSecond, dump a slice:slicedbdump--rootpublic.example'WHERE id IN (7, 56, 234)'--schemaschema.yml>slice.zipThird, restore that slice into another database:slicedbrestore<slice.zipFor a complete working example, seeExample.ConnectionUse thelibpq environment variablesto configure the connection.PGHOST=myhostslicedbschema>slice.ymlDumpSeedump.ymlfor the JSONSchema.Output formatsSliceDB can produce multiple formats:slice- ZIP archive. This can be restored withslicedb restore.sql- SQL file. This can be restored withpsqlor another client. If
restoring into existing schema, foreign keys must first be disabled, e.g.SET session_replication_role = replica.Output contentSchema can optionally be included. Restoring with schema requires an existing
empty database.SchemaTheschemacommand uses foreign keys to infer relationships between tables. It
is a suggested starting point.You may want to prune the slice by removing relationships, or expand the slice
by adding relationships that don't have explicit foreign keys.slicedb schema-filtercan help modify the schema, or generic JSON tools likejq.AlgorithmThe slicing process works as follows:Starting with the root table, query the physical IDs (ctid) of rows.Add the row IDs to the existing list.For new IDs, process each of the adjacent tables, using them as the current
root.Do this in parallel, usingpg_export_snapshot()to guarantee a consistent
snapshot across workers.PerformanceHundreds of thousands of rows can be exported in only a few minutes and several
dozen MBs of memory.TransformationSeetransform.ymlfor the JSONSchema.Replacements are deterministic for a given pepper. By default, the pepper is
randomly generated each run. You may specify it as--pepper. Note that
possession of the pepper makes the data guessable.Transformation may operate on an existing slice (TODO), or happen during the
dump.ConfigurationTransforms are specified by:class, the Python classconfig, transform-specific optionsmodule, defaults toslice_db.transformsThe name given to the transform is appended to the global pepper.Custom transformsTo create custom transforms, implementslice_db.transform.Transform, expose
the class on a module, and install the module so that is accessible byslicedb.Built-in transformsTheslice_db.transformspackage has many common transforms.Seetransforms.mdfor the full list.RestoreSliceDB can restore slices into existing databases. In practice, this should
normally be an empty existing database.CyclesForeign keys may form a cycle only if at least one foreign key in the cycle is
deferrable.That foreign key will be deferred during restore.A restore may happen in a single transaction or not. Parallelism requires
multiple transactions.Not supportedMultiple databasesDatabases other than PostgreSQLDevelopingInstall:make installFormat:make formatPublishingUpdate slice_db/version.pyCommitgit add slice_db/version.py && git commit -m "Version <version>"Taggit tag v<version>Pushgit push origin master --tagsPublish to PyPI:make upload