Package: jsongraph
# jsongraph [![Build Status](https://travis-ci.org/pudo/jsongraph.svg?branch=master)](https://travis-ci.org/pudo/jsongraph)This library provides tools to integrate data from multiple sources into acoherent data model. Given a heterogeneous set of source records, it willgenerate a set of composite entities with merged information from allavailable sources. Further, it allows querying the resulting graph using asimple, JSON-based graph query language.The intent of this tool is to make a graph-based data integration system(based on RDF) seamlessly available through simple JSON objects.## UsageThis is what using the library looks like in a simplified scenario:```pythonfrom jsongraph import Graph# Create a graph for all project information. This can be backed by a# triple store or an in-memory construct.graph = Graph(base_uri='file:///path/to/schema/files')graph.register('person', 'person_schema.json')# Load data about a person.context = graph.context()context.add('person', data)context.save()# Repeat data loading for a variety of source files.# This will integrate data from all source files into a single representation# of the data.context = graph.consolidate('urn:prod')# Metaweb-style queries:for item in context.query([{"name": None, "limit": 5}]):print item['name']```## DesignA ``jsongraph`` application is focussed on a ``Graph``, which stores a set ofdata. A ``Graph`` can either exist only in memory, or be stored in a backenddatabase.All data in a ``Graph`` is structured as collections of JSON objects (i.e.nested dictionaries, lists and values). The structure of all stored objectsmust be defined using a [JSON Schema](http://json-schema.org/). Some limitsapply to such schema, e.g. they may not allow additional or pattern properties.### Contexts and MetadataThe objects in each ``Graph`` are grouped into a set of ``Contexts``. Thosealso include metadata, such as the source of the data, and the level of trustthat the system shall have in those data. A ``Context`` will usually correspondto a source data file, or a user interaction.### Consolidated ContextsWhen working with ``jsongraph``, a user will first load data into a variety of``Contexts``. They can then generate a consolidated version of the data, in aseparate ``Context``.This consolidated version applies entity de-duplication. For object propertieswith multiple available values across several ``Contexts``, the informationfrom the most trustworthy ``Context`` will be chosen.### Queries``jsongraph`` includes a query language implementation, which is heavilyinspired by Google's [Metaweb Query Language](http://mql.freebaseapps.com/ch03.html).Queries are written as JSON, and search proceeds by example. Searches can alsobe deeply nested, traversing the links between objects stored in the ``Graph``at an arbitrary complexity.Queries on the data can be run either against any of the source ``Contexts``,or against the consolidated context. Queries against the consolidated``Context`` will produce responses which reflect the best available informationbased on data from a variety of sources.### De-duplicationOne key part of the functions of this library will be the application ofde-duplication rules. This will take place in three steps:* Generating a set of de-duplicating candidates for all entities in a given``Graph``. These will be simplified representations of objects which can befed into a comparison tool (either automated or interactive with the user).* Once the candidates have been decided, they are transformed into a mapping ofthe type (``original_fingerprint`` -> ``same_as_fingerprint``). Such mappingsare applied to a context.* Upon graph consolidation (see above), the entities which have been mapped toanother are not included. All their properties are inherited by thedestination entity.A data comparison candidate may look like this:```json{"fingerprint": "...","entity": "...","data": {},"source": {"label": "...","url": "http://..."}}```## TestsThe test suite will usually be executed in it's own ``virtualenv`` and perform acoverage check as well as the tests. To execute on a system with ``virtualenv``and ``make`` installed, type:```bash$ make test```