Package: datafreeze
datafreezecreates static extracts of SQL databases for use in interactive
web applications. SQL databases are a great way to manage relational data, but
exposing them on the web to drive data apps can be cumbersome. Often, the
capacities of a proper database are not actually required, a few static JSON
files and a bit of JavaScript can have the same effect. Still, exporting JSON
by hand (or with a custom script) can also become a messy process.Withdatafreeze, exports are scripted in a Makefile-like description,
making them simple to repeat and replicate.InstallationThe easiest way to installdatafreezeis to retrieve it from the Python
package index usingpip:pipinstalldatafreezeBasic UsageCalling DataFreeze is simple, the application is called with a
freeze file as its argument:datafreezeFreezefile.yamlFreeze files can be either written in JSON or in YAML. The database URI
indicated in the Freezefile can also be overridden via the command line:datafreeze--dbsqlite:///foo.dbFreezefile.yamlExample Freezefile.yamlA freeze file is composed of a set of scripted queries and
specifications on how their output is to be handled. An example could look
like this:common:database:"postgresql://user:password@localhost/operational_database"prefix:my_project/dumps/format:jsonexports:-query:"SELECTid,title,dateFROMevents"filename:"index.json"-query:"SELECTid,title,date,countryFROMevents"filename:"countries/{{country}}.csv"format:csv-query:"SELECT*FROMevents"filename:"events/{{id}}.json"mode:item-query:"SELECT*FROMevents"filename:"all.json"format:tabsonAn identical JSON configuration can be found in this repository.Options in detailThe freeze file has two main sections,commonandexports. Both
accept many of the same arguments, withexportsspecifying a list of
exports whilecommondefines some shared properties, such as the
database connection string.The following options are recognized:databaseis a database URI, including the database type, username
and password, hostname and database name. Valid database types includesqlite,mysqlandpostgresql(requires psycopg2).prefixspecifies a common root directory for all extracted files.formatidentifies the format to be generated,csv,jsonandtabsonare supported.tabsonis a condensed JSON
representation in which rows are not represented by objects but by
lists of values.queryneeds to be a valid SQL statement. All selected fields will
become keys or columns in the output, so it may make sense to define
proper aliases if any overlap is to be expected.modespecifies whether the query output is to be combined into a
single file (list) or whether a file should be generated for each
result row (item).filenameis the output file name, appended toprefix. All
occurences of{{field}}are expanded to a fields value to allow the
generation of file names e.g. by primary key. In list mode, templating
can be used to group records into several buckets, e.g. by country or
category.wrapcan be used to specify whether the output should be wrapped
in aresultshash in JSON output. This defaults totrueforlist-mode output andfalseforitem-mode.Contributorsdatasetis written and maintained byFriedrich Lindenberg,Gregor AischandStefan Wehrmeyer. We’re standing on theshoulders of giants.