Package: cubicweb-dataio
Cube for data input/output, import and exportMassive StoreThe Massive Store is a CW store used to push massive amount
of data using pure SQL logic, thus avoiding CW checks.
It is faster than other CW stores (it does not check eid at each step,
it use COPY FROM method), but is less safe (no data integrity securities),
and does not return an eid while using create_entity function.WARNING: This store may be only used with PostgreSQL for now, as it
relies on the COPY FROM method, and on specific PostgreSQL tables
to get all the indexes.Workflow of Massive StoreThe Massive Store workflow is the following:Drop indexes and constraints from the meta-data tables (entities, is_instance_of, …);Insertion of data:using thecreate_entityfunction for entities;using therelatefunction for relations;using therelated_by_iidfunction for relations based on external identifiers;each insertion of a rtype that has not been seen yet will trigger the
creation of a temporary table for this rtype, to store the results.each insertion of an etype that has not been seen yet will remove all
the indexes/constraints on the entity table.At a given point, one should call theflushmethod:it will flush the entities data into the database based onCOPY_FROM.it will flush the relations data into the database based onCOPY_FROM.it will flush the relations-iid data into the database based onCOPY_FROM.it will create the metadata (entities, …) for the insered entities.it will commit.If some relations are created based on external identifiers (relate_by_iid),
the conversion should be manually done using theconvert_relationsmethod.At the end of the insertion, one should call thecleanupmethod:it will re-create the indexes/constraints/primary key for the entities/relations tables.it will re-create the indexes/constraints on the meta-data tables.it will remove temporary tables and internal store tables.Entities/Relations in Massive StoreDue to the technical constraints on the database insertion, there are some following specific points to notice:acreate_entitycall will return an entity with a specificeid. Eids are automatically dealt with
by the Massive Store (it will fetch for a given range of eids for its internal use), but you can
pass a specific eid in the kwargs of thecreate_entitycall to bypass the automatic assignation of an eid.inlined-relations are not supported in therelatemethod.A buffer will be created for the call to the PostgreSQLCOPY_FROMclause.
If the separator used for the creation of this tabular file is found in the data of the entities (or relations),
it will be replace by thereplace_sepof the store (default is to ‘’).Basic use of Massive StoreA simple script using the Massive Store:# Initialize the store
store = MassiveObjectStore(session)
# Initialize the Relation table
store.init_rtype_table('Person', 'lives', 'Location')

# Import logic
...
entity = store.create_entity('Person', ...)
entity = store.create_entity('Location', ...)

# Flush the data in memory to sql database
store.flush()

# Import logic
...
entity = store.create_entity('Person', ...)
entity = store.create_entity('Location', ...)
# Person_iid and location_iid are unique iid that are data dependant (e.g URI)
store.relate_by_iid(person_iid, 'lives', location_iid)
...

# Flush the data in memory to sql database
store.flush()

# Convert the relation
store.convert_relations('Person', 'lives', 'Location')

# Clean the store / rebuild indexes
store.cleanup()In this case,  iid_subj and iid_obj represent an unique id
(e.g. uri, or id from the imported database) that can be used to create
relations after importing entities.Advanced use of Massive StoreThe simple and default use of the Massive Store is conservative to
avoid issues in meta-data management. However it is possible to
increase insertion speed:the flushing of meta-data could be costly if done too many times.
A good practive is to do only once at the end of the import.
For doing so, you should setautoflush_metadatato False in the store creation,
and you should call theflush_meta_dataat the end of the import
(but before the call to `cleanup`).you may avoid to commit at each flush, by settingcommit_at_flushto False in the store creation.  Thus you should explicitely call
thecommitmethod at least oncebefore flushing the meta data
and cleaning up the store.you could avoid dropping the different indexes and constraints using
thedrop_indexattribute during the store creation.you could set a different starting point of the eids sequence using
theeids_seq_startattribute during the store creation.additional callbacks could be given to deal with commit and rollback
(on_commit_callbackandon_rollback_callback).Example of advanced use of Massive Store:store = MassiveObjectStore(session,
                           autoflush_metadata=False,
                           commit_at_flush=False)
store.init_rtype_table('Location', 'names', 'LocationName')
for ind, infos in enumerate(ucsvreader(open(dumpname))):
    entity = {'name': infos[1], ...}
    entity['my_inlined_relation'] =  my_dict.get(infos[2])
    entity = store.create_entity('Location', **entity)
    store.relate_by_iid(entity.cwuri, 'my_external_relation', infos[3])
    if ind and ind % 200000 == 0:
        store.flush()
        store.commit()
store.flush()
store.commit()
store.flush_meta_data()
store.convert_relations('Location', 'my_external_relation', 'Location',
                        'cwuri', 'cwuri')
store.cleanup()Restoring a database after Massive Store failureThe Massive Store remove some constraints and indexes that are
automatically rebuild during thecleanupcall. If there is an error
during the import process, you could still call to thecleanupmethod, or even recreate after the failure another store and call thecleanupmethod of this store.The Massive Store create the following tables for its internal use:dataio_initialized: information on the initialized etype/rtype tables.dataio_constraints: the queries that may be used to restore the constraints/indexes
for the different etype/rtype tables.dataio_metadata: the etypes that have already have their meta-data pushed.Slave ModeA slave mode is available for parallel use of the Massive Store:a Massive Store (master) should be created.for all the possible etype/rtype that may be encoutered during the
import, theinit_etype_table/init_relation_tablemethods of themasterstore should be called.differentslavestores could be created using theslave_modeattribute during the store creation. Theautoflush_metadataattribute should be setted to False.eachslavestore could be used in a different thread, for creating
entity and relation, and should only call to itsflushandcommitmethods.Themasterstore should call itsflush_meta_dataandcleanupmethods at the end of the import.RDF StoreThe RDF Store is used to import RDF data into a CubicWeb data, based
on a Yams <-> RDF schema conversion. The conversion rules are stored
in a XY structure.Building an XY structureYou have to create a file (usually calledxy.py) in your cube, and
import the dataio version of xy:from cubes.dataio import xyYou have to register the different prefixes (common prefixes as skos
or foaf are already registered):xy.register_prefix('diseasome', 'http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseasome/')By default, the entity type is based on the rdf property “rdf:type”,
but you may changed it using:xy.register_rdf_etype_property('skos:inScheme')It is also possible to give a specific callback to determine the
entity type from the rdf properties:def _rameau_etype_callback(rdf_properties):
    if 'skos:inScheme' in rdf_properties and 'skos:prefLabel' in rdf_properties:
       return 'Rameau'

xy.register_etype_callback(_rameau_etype_callback)The URI is fetched from the “rdf:about” property, and can be
normalized using a specific callback:def normalize_uri(uri):
    if uri.endswith('.rdf'):
       return uri[:-4]
    return uri

xy.register_uri_conversion_callback(normalize_uri)Defining the conversion rulesThen, you may write the conversion rules:xy.add_equivalence allows you to add a basic equivalence between
entity type / attribute / relations, and RDF properties. You may use
“*” as a wild cart in the Yams part.  E.g. for entity types:xy.add_equivalence('Gene', 'diseasome:genes')
xy.add_equivalence('Disease', 'diseasome:diseases')E.g. for attributes:xy.add_equivalence('* name', 'diseasome:name')
xy.add_equivalence('* label', 'rdfs:label')
xy.add_equivalence('* label', 'diseasome:label')
xy.add_equivalence('* class_degree', 'diseasome:classDegree')
xy.add_equivalence('* size', 'diseasome:size')E.g. for relations:xy.add_equivalence('Disease close_match ExternalUri', 'diseasome:classes')
xy.add_equivalence('Disease subtype_of Disease', 'diseasome:diseaseSubtypeOf')
xy.add_equivalence('Disease associated_genes Gene', 'diseasome:associatedGene')
xy.add_equivalence('Disease chromosomal_location ExternalUri', 'diseasome:chromosomalLocation')
xy.add_equivalence('* sameas ExternalUri', 'owl:sameAs')
xy.add_equivalence('Gene gene_id ExternalUri', 'diseasome:geneId')
xy.add_equivalence('Gene bio2rdf_symbol ExternalUri', 'diseasome:bio2rdfSymbol')A base URI can be given to automatically determine if a Resource should be considered
as an external URI or an internal relation:xy.register_base_uri('http://www4.wiwiss.fu-berlin.de/diseasome/resource/')A more complex logic can be used by giving a specific callback:def externaluri_callback(uri):
    if uri.startswith('http://www4.wiwiss.fu-berlin.de/diseasome/resource/'):
       if uri.endswith('disease') or uri.endswith('gene'):
          return False
       return True
    return True

xy.register_externaluri_callback(externaluri_callback)The values of attributes are built based on the Yams type. But you
could use a specific callback to compute the correct values from the
rdf properties:def _convert_date(_object, datetime_format='%Y-%m-%d'):
    """ Convert an rdf value to a date """
    try:
       return datetime.strptime(_object.format(), datetime_format)
    except:
       return None

xy.register_attribute_callback('Date', _convert_date)or:def format_isbn(rdf_properties):
    if 'bnf-onto:isbn' in rdf_properties:
       isbn = rdf_properties['bnf-onto:isbn'][0]
       isbn = [i for i in isbn if i in '0123456789']
       return int(''.join(isbn)) if isbn else None

xy.register_attribute_callback('Manifestation formatted_isbn', format_isbn)Importing dataData may thus be imported using the “import-rdf” command of cubicweb-ctl:cubicweb-ctl import-rdf <my-instance> <filer-or-folder>The default library used for reading the data is “rdflib” but one may
use “librdf” using the “–lib” option.It is also possible to force the rdf-format (it is automatically
determined, but this may sometimes lead to errors), using the
“–rdf-format” option.Exporting dataThe view ‘rdf’ may be called and will create a RDF file from the
result set. It is a modified version of the CubicWeb RDFView, that
take into account the more complex conversion rules from the dataio
cube.  The format can also be forced (default is XML) using the
“–format” option in the url (xml, n3 or nt).ExamplesExamples of use of dataio rdf import could be found in the nytimes and
diseasome cubes.