Package: babbage
Babbage Analytical Enginebabbageis a lightweight implementation of an OLAP-style database
query tool for PostgreSQL. Given a database schema and a logical model
of the data, it can be used to perform analytical queries against that
data - programmatically or via a web API.It is heavily inspired byCubesbut
has less ambitious goals, i.e. no pre-computation of aggregates, or
multiple storage backends.babbageis not specific to government finances, and could easily be used e.g. for ReGENESIS, a project that makes German national statistics available via an API. The API functions by interpreting modelling metadata generated by the user (measures and dimensions).Installation and testbabbagewill normally included as a PyPI dependency, or installed viapip:$pipinstallbabbagePeople interested in contributing to the package should instead check out the
source repository and then use the providedMakefileto install the
library (this requiresvirtualenvto be installed):$gitclonehttps://github.com/openspending/babbage.git
$cdbabbage
$makeinstall
$pipinstalltox
$exportBABBAGE_TEST_DB=postgresql://postgres@localhost:5432/postgres
$maketestUsagebabbageis used to query a set of existing database tables, using an
abstract, logical model to query them. A sample of a logical model can be
found intests/fixtures/models/cra.json, and a JSON schema specifying
the model is available inbabbage/schema/model.json.The central unit ofbabbageis aCube, i.e. aOLAP cubethat uses the provided model metadata to construct queries
against a database table. Additionally, the application supports managing
multiple cubes at the same time via aCubeManager, which can be
subclassed to enable application-specific ways of defining cubes and where
their metadata is stored.Futher,babbageincludes a Flask Blueprint that can be used to expose
a standard API via HTTP. This API is consumed by the JavaScriptbabbage.uipackage and it is very closely modelled on the Cubes and OpenSpending HTTP
APIs.Programmatic usageLet's assume you have an existing database table of procurement data and
want to query it usingbabbagein a Python shell. A session might look
like this:importjsonfromsqlalchemyimportcreate_enginefrombabbage.cubeimportCubefrombabbage.modelimportMeasureengine=create_engine('postgresql://localhost/procurement')model=json.load(open('procurement_model.json','r'))cube=Cube(engine,'procurement',model)facts=cube.facts(page_size=5)# There are 17201 rows in the table:assertfacts['total_fact_count']==17201# There's a field called 'total_value':assert'total_value'infacts['fields']# We can get metadata about it:concept=cube.model['total_value']assertisinstance(concept,Measure)assertconcept.label=='Total Value'# And there's some actual data:assertlen(facts['data'])==5fact_0=facts['data'][0]assert'total_value'infact_0# For dimensions, we can get all the distinct values:members=cube.members('supplier',cut='year:2015',page_size=500)assertlen(members['data'])<=500assertmembers['total_member_count']# And, finally, we can aggregate by specific dimensions:aggregate=cube.aggregate(aggregates='total_value.sum',drilldowns='supplier|authority'cut='year:2015|authority.country:GB',page_size=500)# This translates to:#   Aggregate the procurement data by summing up the 'total_value'#   for each unique pair of values in the 'supplier' and 'authority'#   dimensions, and filter for only those entries where the 'year'#   dimensions key attribute is '2015' and the 'authority' dimensions#   'country' attribute is 'GB'. Return the first 500 results.assertaggregate['total_cell_count']assertlen(aggregate['cells'])<=500aggregate_0=aggregate['cells'][0]assert'total_value.sum'inaggregate_0# Note that these attribute names are made up for this example, they# should be reflected from the model:assert'supplier.code'inaggregate_0assert'supplier.label'inaggregate_0assert'authority.code'inaggregate_0assert'authority.label'inaggregate_0Using the HTTP APIThe HTTP API forbabbageis a simple FlaskBlueprintused to expose a small set of calls that correspond to
the cube functions listed above. To include it into an existing Flask
application, you would need to create aCubeManagerand then
configure the API like this:fromflaskimportFlaskfromsqlalchemyimportcreate_enginefrombabbage.managerimportJSONCubeManagerfrombabbage.apiimportconfigure_apiapp=Flask('demo')engine=models_directory='models/'manager=JSONCubeManager(engine,models_directory)blueprint=configure_api(app,manager)app.register_blueprint(blueprint,url_prefix='/api/babbage')app.run()Of course, you can define your ownCubeManager, for example if
you wish to retrieve model metadata from a database.When enabled, the API will expose a number of JSON(P) endpoints
relative to the givenurl_prefix:/, returns the system status and version./cubes, returns a list of the available cubes (name only)./cubes/<name>/model, returns full metadata for a given
cube (i.e. measures, dimensions, aggregates etc.)/cubes/<name>/factsis used to return individual entries from
the cube in a non-aggregated form. Supports filters (cut), a
set offieldsto return and asort(field_name:direction),
as well aspageandpage_size./cubes/<name>/membersis used to return the distinct set of
values for a given dimension, e.g. all the suppliers mentioned in
a procurement dataset. Supports filters (cut), a and asort(field_name:direction), as well aspageandpage_size./cubes/<name>/aggregateis the main endpoint for generating
aggregate views of the data. Supports specifying theaggregatesto include, thedrilldownsto aggregate by, a set of filters
(cut), a and asort(field_name:direction), as well
aspageandpage_size.