Package: dbcut
DBcutDBcut aims to allow the extraction of lightweight subset of relational production database for development and testing
purpose.Table of ContentsOverviewUsageGetting startedUnder The HoodDatabase Reflection and Loading
StategySQL from YAMLExtraction GraphOverviewIts main features are:Extract data from large databases.Reinject data into another base.Target and source databases could be based on different DBMS (i.e., MySQL -> PostgreSQL/SQLite).Extraction queries simplified in YAML.Support nested associations.Json and plain SQL export.Reasonable performance.Caching of extractions to accelerate future extractions.UsageUsage:dbcut[OPTIONS]COMMAND1[ARGS]...[COMMAND2[ARGS]...]...ExtractalightweightsubsetofyourproductionDBfordevelopmentandtestingpurpose.Options:-c,--configPATHConfigurationfile--versionShowtheversionandexit.-y,--force-yesNeverpromptsforuserintervention-i,--interactivePromptsforuserintervention.--quiet,--no-quietSuppressesmostwarninganddiagnosticmessages.--debugEnablesdebugmode.--verboseEnablesverboseoutput.-h,--helpShowthismessageandexit.Commands:loadExtractandloaddatatothetargetdatabase.flushRemoveALLTABLESfromthetargetdatabaseandrecreatetheminspectCheckdatabasescontent.dumpsqlDumpallSQLinsertqueries.dumpjsonExportdatatojson.clearRemovealldata(only)fromthetargetdatabasepurgecacheRemoveallcachedqueries.Getting startedLet’s take the following database example:We want to extract some users with all related data to our development database.First, we have to edit the extraction filedbcut.yamlas follows:# dbcut.ymldatabases:source_uri:mysql://foo:bar@db-host/proddestination_uri:sqlite:///small-dev-database.dbqueries:-from:userlimit:2Then, we set the limit to two users, the default limit being 10.After that, we launch the extraction command with theloadcommand:$dbcutload--->Reflectingdatabaseschemafrommysql://foo:***@db-host/prod--->Creatingnewsqlite:///small-dev-database.dbdatabase--->Creatingalltablesandrelationsonsqlite:///small-dev-database.dbQuery1/1:from:userlimit:2backref_limit:10backref_depth:5join_depth:5exclude:[]include:[]┌─ⁿ─comment├─ⁿ─voteuser┤└─ⁿ─user_group┐└─¹─group┐└─¹─role┐└─ⁿ─role_permission┐└─¹─permission8tablesloaded--->Cachekey:4a468c3555074890b7c342c0a575f29d47145821--->Executingquery--->Fetchingobjects--->Inserting31rowsWe can check the data on our new database :$lsdbcut.ymlsmall-dev-database.db$sqlite3small-dev-database.dbsqlite>SELECTid,loginFROMuser;3|jerome4|juliensqlite>SELECT*fromcomment;8|commentjerome1|39|commentjerome2|310|commentjerome3|3In the following example, we are going to retrieve roles with related groups and permissions. In order to obtain the
best extraction graph, we are going to use the keywordinclude, which indicated to dbcut that we want to minimize
the number of associated tables (Nested associations).queries:-from:userlimit:2-from:roleinclude:-group-permissionIt is possible to empty the content of the local database before beginning the extraction with theclearcommand.$dbcut-yclearload--->Removingalldatafromsqlite:///small-dev-database.dbdatabase--->Reflectingdatabaseschemafrommysql://foo:***@db-host/prod?charset=utf8--->Creatingalltablesandrelationsonsqlite:///small-dev-database.dbQuery1/2:from:userlimit:2backref_limit:10backref_depth:5join_depth:5exclude:[]include:[]┌─ⁿ─comment├─ⁿ─voteuser┤└─ⁿ─user_group┐└─¹─group┐└─¹─role┐└─ⁿ─role_permission┐└─¹─permission8tablesloaded--->Cachekey:4a468c3555074890b7c342c0a575f29d47145821--->Usingcache(2elements)--->Fetchingobjects--->Inserting31rowsQuery2/2:from:rolelimit:10backref_limit:10backref_depth:nulljoin_depth:nullexclude:[]include:-group-permission┌─ⁿ─grouprole┤└─ⁿ─role_permission┐└─¹─permission4tablesloaded--->Cachekey:5029d84dbb2bc75a7df898dd94df93b395e91e44--->Executingquery--->Fetchingobjects--->Inserting22rowsAs you can see in the first query, the cache was used and there was thus no interaction with the source database.This query allowed the extraction of all roles:sqlite>SELECT*fromrole;1|admin2|moderator3|userIf we had not used theincludekeyword, all tables would have been extracted:┌─ⁿ─role_permission┐
    │                  └─¹─permission
role┤
    └─ⁿ─group┐
             └─ⁿ─user_group┐
                           │       ┌─ⁿ─comment
                           └─¹─user┤
                                   └─ⁿ─voteTo narrow more precisely our extraction, we are now going to limit to roles that can delete a user.queries:-from:userlimit:2-from:roleinclude:-group-permissionwhere:permission.codename:'delete_user'Only the last extraction rule is relaunched with the--last-onlyoption.$dbcut-yclearload--last-only...--->Cachekey:ffb664a2e69c88fa48db2680daf71d30408bd207--->Executingquery--->Fetchingobjects--->Inserting14rowsThis time, only the ‘admin’ role is retrieved:sqlite>SELECT*fromrole;1|adminPlease note that the filter only applies here to role table (from) and not to the permission.sqlite>SELECT*FROMpermission";1|delete_comment2|delete_vote3|delete_user4|create_comment5|create_vote6|create_userIndeed, we filter the roles based on a value from the permission table, but we do retrieved all permissions associated
to this role.In the above example, it makes sense that the admin role has all permissions.Last but not least, we can also retrieve data in json or raw sql format !$dbcutdumpjson|dumpsql[{"password":"julien","vote_collection":[{"user_id":4,"comment_id":1,"id":3,"rating":4},{"user_id":4,"comment_id":3,"id":6,"rating":10},{"user_id":4,"comment_id":6,"id":13,"rating":10}],"comment_collection":[],"id":4,"login":"julien","user_group_collection":[{"user_id":4,"group":{"name":"Utilisateur","role":{"id":3,"role_permission_collection":[{"permission":{"id":4,"codename":"create_comment","role_permission_collection":[]},PRAGMAforeign_keys=OFF;BEGIN;INSERTORIGNOREINTOpermission(id,codename)VALUES(4,'create_comment');INSERTORIGNOREINTOpermission(id,codename)VALUES(5,'create_vote');INSERTORIGNOREINTOpermission(id,codename)VALUES(1,'delete_comment');INSERTORIGNOREINTOpermission(id,codename)VALUES(2,'delete_vote');INSERTORIGNOREINTOrole(id,name)VALUES(3,'user');INSERTORIGNOREINTOrole(id,name)VALUES(2,'moderator');INSERTORIGNOREINTOuser(id,login,password)VALUES(4,'julien','julien');INSERTORIGNOREINTOuser(id,login,password)VALUES(3,'jerome','jerome');INSERTORIGNOREINTO"group"(id,name,role_id)VALUES(3,'Utilisateur',3);INSERTORIGNOREINTO"group"(id,name,role_id)VALUES(2,'Moderateur',2);INSERTORIGNOREINTOcomment(id,content,user_id)VALUES(8,'comment jerome 1',3);INSERTORIGNOREINTOcomment(id,content,user_id)VALUES(9,'comment jerome 2',3);INSERTORIGNOREINTOcomment(id,content,user_id)VALUES(10,'comment jerome 3',3);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(12,3,4);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(13,3,5);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(7,2,4);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(8,2,5);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(10,2,1);INSERTORIGNOREINTOrole_permission(id,role_id,permission_id)VALUES(11,2,2);INSERTORIGNOREINTOuser_group(id,user_id,group_id)VALUES(4,4,3);INSERTORIGNOREINTOuser_group(id,user_id,group_id)VALUES(3,3,2);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(3,4,4,1);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(6,10,4,3);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(13,10,4,6);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(2,5,3,1);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(5,1,3,2);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(7,10,3,3);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(10,6,3,1);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(11,5,3,5);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(12,6,3,6);INSERTORIGNOREINTOvote(id,rating,user_id,comment_id)VALUES(19,10,3,10);COMMIT;Under The HoodDatabase Reflection and Loading StategyDBcut heavily uses SQLAlchemy, the SQL toolkit and Object Relational Mapper for Python. The ORM makes it possible to
free ourselves from the SQL direct manipulation, but that is not all. SQLAlchemy offers a range of toolkits that enable
us to programmatically build all SQL queries useful to DBcut. This include both the schema creation and all of its
properties, the select, join and insert queries… no matter which DBMS is used (PostgreSQL, MySQL, SQLite, oracle etc.).One of the most important features of DBcut is that the user does not need to know or provide the source database
schema to use it. First of all, DBcut will inspect the source database and retrieve all metadata. This action is what we
call:Database Reflection.The MetaData object store all the collection of metadata entities. DBcut will alter this MetaData object to make it
compatible with most DBMS. For example, the names of indexes or foreign keys can be too long for SQLite but not for
MYSQL. Sometimes, it also changes the types of the column to make it match what is expected in the target database.
(mysql.TINYINTbecameSMALLINTin SQLite and PostgreSQL)Once the MetaData object is complete, we can create the new database which is almost identical to the source database
(except some compatibility adjustments)DBcut will generate and launch extraction request on the source database. The data thus obtained will be detached from
the first SQLAlchemy session to be attached to the new session in the target database. This is where the SQLAlchemy
magic happens: the same request will be used to extract data from the source database and to load them into the target
database. Indeed, in the first case (query/fetch), it will be translated into SQLSELECTqueries and in the second
case, into SQLINSERTstatements (load).SQL from YAMLOne of the goals of DBcut is to allow quick writing of extraction requests. Most of the time, to write an extraction
request, not much information is needed: only the main table name, hoping to retrieve the maximum number of related
data as possible.The idea was to find a sufficiently concise syntax that allows us to build the most complete extraction requests with
the minimum effort.The YAML came to us naturally as it is pleasant to read, easy to understand and to edit for humans.Thedbcut.ymlfile is both used to configure DBcut and to write
extraction requests.databases:source_uri:mysql://chinook:chinook@192.168.66.66/chinookdestination_uri:sqlite:///chinook.dbqueries:-from:customer_customerTo write an extraction request, only the keywordfromis mandatory. However, other keywords can be added to reduce
the size of data to retreive.-from:contracts_customerwhere:brand:2limit:100backref_limit:500backref_depth:2join_depth:5exclude:-django_admin_log-django_sessioninclude:[]Unlike the SQL queries, an extraction request using DBcut automatically and recursively loads all associated relations
(SeeExtraction Graph). All these options are filtering and reducing options that prevents from
slowing down the extraction process.Finally, with the scope of making the extraction requests as compact as possible, we can add default values to most of
these options:default_limit:100default_backref_limit:500default_backref_depth:2default_join_depth:5global_exclude:-django_admin_log-django_sessionExtraction GraphTo build an extraction request, we first build its extraction graph.An extraction graph is a subset of the complete graph of database relations. Every node represents a table, and each
link represents a relation between two tables. The link direction is defined by the foreign key.To build this graph, we use theMetaDataobject (SeeDatabase Reflection and Loading Stategy).Let’s use the following database schema:The retrieved metadata during the database reflection are used to build
the following complete graph of relations:To build the extraction graph, we browse the complete graph starting
from the table used in thefrominstruction. The browsing only stops
if :the link has already been browsedthe table is explicitly excludedthe maximum depth is reachedFor the following request:queries:-from:customer_customerThe generated extraction graph is:Please note that we handle the two types of relations : one-to-many relations (noted1in the extraction graph) and
many-to-many relations (notedn).CHANGELOGAll notable changes to this project will be documented in this file.The format is based onKeep a Changelog, and this project adheres toSemantic Versioning.Version 0.6.0Released on April 13th 2021AddedBetter support for tables without primary keyLoad all many-to-many relationships just like one-to-many onesVersion 0.5.0Released on April 09th 2021AddedAdded support of SQLAlchemy 1.4Version 0.4.1Released on April 01st 2021ChangedDowngrade sqlalchemy requirement to <=1.4 versions to prevent crashingVersion 0.4.0Released on March 12th 2021ChangedEnsure that generated names for SQLAlchemy relationships do not conflict with columns namesAlways generate unique indexes on sqliteTranslate mysqlcurrent_timestamp()toCURRENT_TIMESTAMPon sqliteVersion 0.3.1Released on March 05th 2021AddedNew experimental class Recorder that record all SQL interactions in order to replay them offline (for testing purpose)Version 0.2.0Released on August 21st 2020AddedAlways enable SQLAlchemy post_update to avoid circular dependeciesDisable all cache ifcacheconfig key is set tonoDo not globally exclude already loaded relations from the other extraction graph branchesImproved cyclical relations loading in the extraction graphChangedStore cache by dbcut versionRemovedRemoved marshmallow serialization and prefer builtin jsonVersion 0.1.6Released on August 20th 2020Changedflushpurges the cache only when–with-cacheis passedFixedLoad only transient mapper objects to force sqlalchemy to generate sql insert queriesVersion 0.1.5Released on August 11th 2020Changeddumpsqlprints only create and insert sql statementsFixedFixed dumpjson regressionFixed query caching mechanismPrepared all mapper objects correctly when metadata is cachedVarious minor bug fixesVersion 0.1.4Released on May 07th 2020FixedFixed TypeError exceptionDefined a max length for indexes on TEXT column on mysql databasesVersion 0.1.3Released on November 27th 2019Changedclearcmd delete only existing table.FixedDetermistic cache key generation.First release on PyPI.