Package: pycopyql
pycopyqlExports a subset of data from a relational database.Inspired bycopyql. ❤️InstallationUsepip.pipinstallpycopyqlUsagepycopyql[-h][-cCONFIG][-fFORMAT]connectionoutputquery[query...]-h: Shows usage information-c: Path to a configuration file, defaults to./pycopyql.py-f: Output format,sqlandjsonare supported, defaults tosqlconnection: Name of the connection to use, defined in the configuration fileoutput: Path to output filequery: One or more query strings (delimited by spaces) of the formatTABLE.COLUMN:VALUE(e.g.accounts.id:1)Example:pycopyql-c./path/to/pycopyql.py-fjsonmy_connectionpath/to/output.sqlaccounts.id:1users.id:2Configurationpycopyqluses a Python file for configuration. It looks like this.# pycopyql.pyconfig={'connections':{'my_connection':{'drivername':'mysql+mysqlconnector','username':'myuser','password':'mypassword','host':'localhost','port':3306,'database':'mydatabase',}}}The file must define aconfigvariable that is assigned adictionary. That dictionary must have a'connections'key that is likewise assigned a dictionary.Within'connections', each key-value pair represents the configuration for a single database connection. The key is a name used to reference the connection when invokingpycopyql.Under the hood,pycopyqluses the popularSQLAlchemy library. Most settings for each connection correspond to the settings used forSQLAlchemy URLs.ResolversAresolverinpycopyqlis a function that receives the names of a table and a column contained in that table and returns a list of table-column pairs related to the specified table-column pair thatpycopyqlwill then search for data to include in its output.Let's take a look at an example of a resolver that's bundled withpycopyql.❯ python3
Python 3.7.0 (default, Jun 29 2018, 20:14:27)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pycopyql.resolver.inflector import dependents_inflector
>>> help(dependents_inflector)

Help on function dependents_inflector in module pycopyql.resolver.inflector:

dependents_inflector(meta, table, column)
    Uses inflection of column names to return foreign keys from tables that reference the specified column if it is a primary key.

    Parameters:
        meta (sqlalchemy.schema.MetaData): Metadata for the database structure
        table (string): Name of the table containing the column being examined
        column (string): Name of the column being examined

    Returns:
        list: List of dictionaries each containing the table and column of foreign key columns that reference the specified columnLet's assume that we're using this resolver with a database that looks like this.+-----------+     +------------+     +---------+
| libraries <--+  | books      |  +--> authors |
+-----------+  |  +------------+  |  +---------+
| id        |  |  | id         |  |  | id      |
| name      |  +--+ library_id |  |  | name    |
| owner     |     | title      |  |  +---------+
+-----------+     | author_id  +--+
                  +------------+Given a table name of'libraries'and a column name of'id',dependents_inflectorwould return this.[{'table':'books','column':'library_id'},]pycopyql.resolver.inflectoralso includes another resolver calleddependencies_inflector, which works in the opposite direction ofdependents_inflector. That is, wheredependents_inflectorreturns foreign keys that reference a primary key,dependencies_inflectorreturns the primary key corresponding to a foreign key.Using the above database as an example again, given a table name of'books'and a column name of'library_id',dependencies_inflectorwould return this.[{'table':'libraries','column':'id'},]A custom resolver that handles both of these cases using the bundled resolvers shown above might look like this.# pycopyql.pyfrompycopyql.resolver.inflectorimportdependents_inflector,dependencies_inflectordefmy_resolver(meta,table,column):dependents=dependents_inflector(meta,table,column)dependencies=dependencies_inflector(meta,table,column)returndependents+dependenciesconfig={'connections':{'my_connection':{# SQLAlchemy connection parameters go here'resolver':my_resolver,},},}In fact,pycopyql.resolver.inflectorcontains yet another resolver calledinflectorthat functions exactly this way.Custom resolvers can operate in many ways. For example, one might use foreign keys to determine relationships. Thedependents_foreign_keyanddependencies_foreign_keyresolvers contained inpycopyql.resolver.foreign_keyare equivalent to thedependents_inflectoranddependencies_inflectorresolvers described earlier.pycopyql.resolver.foreign_keyalso includes theforeign_keyresolver, which is likewise equivalent to theinflectorresolver described earlier.For some cases where database column naming conventions are inconsistent, or where one table requires multiple references to another table (e.g. identifiers of the last users who created and updated a record), a resolver can use a static list of keys specific to that database. Here's an example using the database schema shown earlier.defmy_resolver(meta,table,column):keys={'libraries':{'id':[{'table':'books','column':'library_id'}],},'books':{'library_id':[{'table':'libraries','column':'id'}],'author_id':[{'table':'authors','column':'id'}]},'authors':{'id':[{'table':'books','column':'author_id'}],},}iftableinkeysandcolumninkeys[table]:returnkeys[table][column]return[]pycopyql.resolver.staticcontains a convenience function,static_map, that takes in a dictionary like the one assigned tokeysin the example above and returns a resolver that will use that static map in the same way.frompycopyql.resolver.staticimportstatic_mapkeys={'table_1':{'column_1':[{'table':'other_table_1','column':'other_column_1'},{'table':'other_table_2','column':'other_column_2'},# ...]},'table_2':{# ...},# ...}config={'connections':{'my_connection':{# ...'resolver':static_map(keys),},},}Custom resolvers can use several of the resolvers described here together simultaneously. For example, if you wish to use inflection, but have some cases for which you must provide a static mapping, you can do both using a custom resolver.frompycopyql.resolver.inflectorimportinflectorfrompycopyql.resolver.staticimportstatic_mapstatic_keys={# ...}static_resolver=static_map(static_keys)defmy_resolver(meta,table,column):static=static_resolver(meta,table,column)inflected=inflector(meta,table,column)returnstatic+inflectedOutput FormatsBy default,pycopyqlsupportssqlandjsonoutput formats. You can add additional formats or replace the bundled export functions for supported formats with your own.In your configuration file, on the same level as the'connections'key, add an'exporters'key. The corresponding value should be a dictionary keyed by a format name that will be passed topycopyqlusing its-fflag. The value should be a function that outputs given data in that format.Let's take a look at one of the bundled export functions.❯ python3
Python 3.7.0 (default, Jun 29 2018, 20:14:27)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pycopyql.export import export_sql
>>> help(export_sql)

Help on function export_sql in module pycopyql.export:

export_sql(meta, data, output)
    Outputs data as SQL INSERT statements.

    Parameters:
        meta (sqlalchemy.schema.MetaData): Metadata for the database structure
        data (dict): Dictionary keyed by table name of dictionaries corresponding to table rows
        output (string): Path for output file

    Returns:
        NoneIf you wanted to override the bundled SQL exporter, here's what it might look like.defmy_sql_exporter(meta,data,output):# ...config={'connections':{# ...},'exporters':{'sql':my_sql_exporter,},}LicenseMIT License