Package: losser
[![Build Status](https://travis-ci.org/ckan/losser.svg)](https://travis-ci.org/ckan/losser)[![Coverage Status](https://img.shields.io/coveralls/ckan/losser.svg)](https://coveralls.io/r/ckan/losser)[![Latest Version](https://pypip.in/version/losser/badge.svg)](https://pypi.python.org/pypi/losser/)[![Downloads](https://pypip.in/download/losser/badge.svg)](https://pypi.python.org/pypi/losser/)[![Supported Python versions](https://pypip.in/py_versions/losser/badge.svg)](https://pypi.python.org/pypi/losser/)[![Development Status](https://pypip.in/status/losser/badge.svg)](https://pypi.python.org/pypi/losser/)[![License](https://pypip.in/license/losser/badge.svg)](https://pypi.python.org/pypi/losser/)Losser======A little UNIX command and Python library for lossy filter, transform, andexport of JSON to Excel-compatible CSV.Created for [ckanapi-exporter](https://github.com/ckan/ckanapi-exporter).Losser can either be run as a UNIX command or used as a Python library(see [Usage](#usage) below). It takes a JSON-formatted list of objects(or a list of Python dicts) as input and produces a "table" as output.The input objects don't all have to have the same fields or structure as eachother, and may contain sub-lists and sub-objects arbitrarily nested.The output "table" is a list of objects that all have the same keys in the sameorder, and with sub-objects and sub-lists nested no more than one level deep.It can be output as:* A list of Python OrderedDicts each having the same keys in the same order* A string of JSON-formatted text representing a list of objects each havingthe same keys in the same order([TODO](https://github.com/ckan/losser/issues/3))* A string of CSV-formatted text, one object per CSV row. The rows of the CSVcorrespond to the objects in the list of output objects if they had beenreturned as Python or JSON Data, and the columns correspond to the objects'keys.The input objects can be filtered and transformed before producing the outputtable. You provide a list of "column query" objects in a `columns.json` filethat specifies what columns the output table should have, and how the valuesfor those columns should be retrieved from the input objects.For example, if you had some input objects that looked like this:[{"author": "Sean Hammond","title": "An Example Input Object","extras":{"Delivery Unit": "Commissioning"{},...]You might transform them using a `columns.json` file like this:{"Data Owner": {"pattern_path": "^author$"},"Title": {"pattern_path": "^title$"},"Delivery Unit": {"pattern_path": ["^extras$", "^Delivery Unit$"]}}This would output a CSV file like this:Data Owner,Title,Delivery UnitSean Hammond,An Example Input Object,CommissioningFrank Black,Another Example Object,Some Other Unit...The `columns.json` file above specifies three column headings for the outputtable:1. Data Owner2. Title3. Delivery UnitThe values for each column are retrieved from the input objects by following a"pattern path": a list of regular expressions that are matched against the keysof the input object and its sub-objects in turn to find a value.For example the "Data Owner" field above has the pattern path `"^author$"` whichmatches the string "author". This will find top-level keys named "author" inthe input objects and output their values in the "Data Owner" column of theoutput table.The "Delivery Unit" column above has a more complex pattern path:`["^extras$", "^Delivery Unit$"]`. This will find the top-level key "extras" inan input object and, assuming the value for the "extras" key is a sub-object,will find and return the value for the "Delivery Unit" key in the sub-object.Pattern paths can be arbitrarily long, recursing into arbitrarily deeply nestedsub-objects.One of the patterns in a pattern path may match multiple keys in an object orsub-object. In that case losser recurses on each of the matched keys and endsup returning a list of values instead of a single value.For example given this input object:{"update": "yearly","update frequency": "monthly",...}The pattern path `"^update.*"` (which matches both "update" and "updatefrequency") would output `"yearly, monthly"` (a quoted, comma-separated list)in the corresponding cell in the CSV output.If a pattern path goes through a sub-list in the input dict losser will iterateover the list and recurse on each of its items. Again it will end up returninga list of values instead of a single value.For example, given a list of input objects like this:[{"resources": [{"format": "CSV",...},{"format": "JSON",...},...],...},...]The pattern path `["^resources$", "^format$"]` will find each object's"resources" sub-list and then find the "format" field in each object in thesub-list. The values in the CSV column will be lists like `"CSV, JSON"`.List can optionally be deduplicated.Nested lists can occur (when the input object contains a list of lists, forexample). These are flattened in the output cells.Some of the filtering and transformations you can do with losser include:* Extract some fields from the objects and filter out others.Any fields in an input object that do not match any of the pattern paths inthe `columns.json` file are filtered out.([TODO](https://github.com/ckan/losser/issues/2): Support appending unmatchedfields to the end of the ouput table as additional columns).* Specify the order of the columns in the output table.Columns are output in the same order that they appear in the `columns.json`file, which does not have to be the same order as the corresponding fields inthe input objects.* Rename fields, using a different name for the column in the output table thanfor the field in the input objects.For example to get the "notes" field from each input object and place themall in a "description" column in the output table, put this object in your`columns.json`:"Description": {"pattern_path": "^notes$",}* Match patterns case-sensitively.By default patterns are matched case-insensitively. To do case-sensitivematching put `"case_sensitive": true` in a column query in your`columns.json` file:"Title": {"pattern_path": "^title$","case_sensitive": true},This will match "title" in the input object, but not "Title" or "TITLE".* Transform the matched values, for example truncating or stripping whitespacefrom strings.* Provide arbitrary pre-processor and post-processor functions to do customtransformations on the input and output objects([TODO](https://github.com/ckan/losser/issues/1)).* Find inconsistently-named fields using a pattern that matches any of thenames and combine them into a single column in the output table.For example you can provide a pattern like `"^update.*"` that will find keysnamed "update", "Update", "Update Frequency" etc. in different input objectsand collect their values in a single "Update Frequency" column.* Collect multiple fields together in a single column.If a pattern matches multiple fields they'll be output as a quotedcomma-separated list in a single cell in the CSV.For example with an input object like this:{"Contributor 1": "Thom Yorke","Contributor 2": "Nigel Godrich","Contributor 3": "Jonny Greenwood",...}The pattern `"^Contributor.*"` will match all three fields and the value inthe CSV cell will be `"Thom Yorke,Nigel Godrich,Jonny Greenwood"`.* You can specify that a pattern path should find a unique value in the object,and if more than one value in the object matches the pattern (and a listwould be returned) an exception will be raised.Use `"unique": true` in a column query in your `columns.json` file:"Title": {"pattern_path": "^title$","unique": true},This is useful for debugging pattern paths that you expect to be unique.* You can specify that a pattern path *must* match a value in the object, andan exception will be raised if there's no matching path through the object([TODO](https://github.com/ckan/losser/issues/4)).* When a pattern matches multiple paths through the input object, or matches apath going through a sub-list, the resulting list of values in the outputtable cell can be deduplicated. Put `"deduplicate": true` in a column queryin your `columns.json` file:"Format": {"pattern_path": ["^resources$", "^format$"],"deduplicate": true},What it can't do (yet):* Pattern match against the values of items (as opposed to their keys).When following a pattern path through an object, when losser hits anobject/dictionary in the input, either one of the top-level objects in thelist of input objects or a sub-object, losser matches the relevant regexagainst the object's keys and then recurses on the values of each of thematched keys.If the key matches the pattern it recurses, you can't also specify a patternto match the value against.When it hits a string, number, boolean or ``None``/``null`` losser returnsit. You can't give it a pattern to match the value against to decide whetherto return it or not.When it hits a list losser iterates over the items in the list and for eachitem either returns it or, if it's a sub-list or sub-object, recurses.(When sub-lists or sub-objects would cause a nested list to be returned it'sflattened into a single list and optionally deduplicated.) Again, you can'tprovide a pattern to be matched against each item to decide whether toreturn/recurse or not.Adding pattern matching against values as well as keys would add a lot ofpower.Requirements------------Python 2.7.Installation------------To install run:pip install losserTo install for development, create and activate a Python virtual environmentthen do:git clone https://github.com/ckan/losser.gitcd losserpython setup.py developpip install -r dev-requirements.txtUsage-----On the command-line losser reads input objects from stdin and writes the outputtable to stdout, making it composable with other UNIX commands. For example:losser --columns columns.json < input.json > output.csvYou can also specify columns on the command line instead of using acolumns.json file. For example:losser --column "Data Owner" --pattern '^author$' --unique --case-sensitive --column "Description" --pattern '^notes$' --unique --case-sensitive --max-length 255 --column Formats --pattern '^resources$' '^format$' --case-sensitive --deduplicateYou specify one or more `--column` options with the column title as argumentand followed by a `--pattern` option and any other column options (`--unique`,`--case-sensitive`, etc).The `--pattern` option can take more than one space-separated argument, in thecase where the column's pattern path contains more than one pattern, forexample: `--pattern "^resources$" "^format$"`.Column options like `--pattern`, `--unique`, `--max-length` etc apply to thepreceding `--column` on the command-line.See `losser -h` for help.This will read input objects from `input.json`, read column queries from`columns.json`, and write output objects to `output.csv`.To use losser as a Python library:import losser.losser as lossertable = losser.table(input_objects, columns)`input_objects` should be a list of dicts. `columns` can be either a list ofdicts or the path to a `columns.json` file (string). The returned `table` willbe a list of dicts. If you pass `csv=True` to `table()` it'll return aCSV-formatted string instead. See `table()`'s docstring for more arguments.Inheriting Losser's Command Line Interface------------------------------------------Losser's command line interface with `--column` and related arguments is fairlycomplicated to implement. You may want to offer the same command line featuresin your own losser-based command without having to reimplement them.For example [ckanapi-exporter](https://github.com/ckan/ckanapi-exporter) offersthe losser command line interface but adds its own `--url` and `--apikey`arguments to pull the input data directly from a CKAN site instead of readingit from stdin.`losser.cli` provides `make_parser()` and `parse()` functions to enableinheriting its command-line interface. Here's how you use them:parent_parser = losser.cli.make_parser(add_help=False, exclude_args=["-i"])parser = argparse.ArgumentParser(description="Export datasets from a CKAN site to JSON or CSV.",parents=[parent_parser],)parser.add_argument("--url", required=True)parser.add_argument("--apikey")try:parsed_args = losser.cli.parse(parser=parser)except losser.cli.CommandLineExit as err:sys.exit(err.code)except losser.cli.CommandLineError as err:if err.message:parser.error(err.message)url = parsed_args.urlcolumns = parsed_args.columnsapikey = parsed_args.apikeydatasets = get_datasets_from_ckan(url, apikey)csv_string = losser.losser.table(datasets, columns, csv=True)See [ckanapi-exporter](https://github.com/ckan/ckanapi-exporter) for aworking example.Running the Tests-----------------First activate your virtualenv then install the dev requirements:pip install -r dev-requirements.txtThen to run the tests do:nosetestsTo run the tests and produce a test coverage report do:nosetests --with-coverage --cover-inclusive --cover-erase --cover-tests