Package: parscival
DescriptionParscival is a data parsing and transformation tool. Even though it can potentially
interpret any input format and subsequently produce any output format, it was originally
designed to process PubMed.nbibfiles and export them to the CorText Graph format.Data parsing and transforming is performed according to an experimental specification
described in a YAML file. For an example seehere.The output parsed data is saved by default using theHDF5binary data format. HDF5
is an open source file format that supports large, complex, heterogeneous data.
It is designed for fast I/O processing and storage.To enable parallel (on-the-fly) access to the HDF5 data produced, Parscival
usesklepto, a python library that
provides fast and flexible access to large amounts of storage.In order to define how to transform the parsed data into an arbitrary output
format, Parscival implements a lightweight plugin architecture. For example, by using
therender-templateplugin, the output
result can be simple described as aJinjatemplate. For an example on how to transform the parsed data intojsonseehere.Installpip install parscivalUsageusage: parscival [-h] [--version] [-v] [-vv] FILE_PARSER_SPEC FILE_OUTPUT FILE_DATASET [FILE_DATASET ...]A modular framework for parsing, mapping and transforming datapositional arguments:FILE_PARSER_SPEC     parser specificationFILE_OUTPUT          parsed data outputFILE_DATASET         input datasetoptional arguments:-h, --help           show this help message and exit--version            show program's version number and exit-v, --verbose        set loglevel to INFO-vv, --very-verbose  set loglevel to DEBUGExamples#convertsdocumentsfrompesticides-s.nbibintopesticides.cortext.jsonasdescribedbypubmed-nbib.yamlparscival -v src/parscival_specs/pubmed-nbib.yaml /tmp/pesticides.cortext.json tests/datasets/pesticides-s.nbib#convertsdocumentsfrombothpesticides-s.nbibandhetercat-s.nbibintopesticides.dbasdescribedbypubmed-nbib.yamlparscival -v src/parscival_specs/pubmed-nbib.yaml /tmp/pesticides.cortext.db tests/datasets/pesticides-s.nbib tests/datasets/hetercat-s.nbibSupported formatsInputPubMed nbib: PubMed is a free search engine accessing primarily the MEDLINE
database of references and abstracts on life sciences and biomedical topics. The
parsing spec is avalaiblehere. You can find a more
detailed description in the relateddocumentation.ParsingThe intermediate parsing data is stored usign the CorText Graph format:FieldValueTypeDescriptionfilesourceFile(fieldName)textsource file for the dataidfieldName.doc[0,n-1]integerID of each documentrankfieldName.doc[id][0,m-1]integerfield cardinal indexparserankfieldName.doc[id][rank][0,p-1]integerparsed cardinal indexdatafieldName.doc[id][rank][parserank][text,integer]parsed dataOutputcortext.json: Parsed data is converted tojsonusing thecortext.json templatecortext.sqlite: Parsed data is converted to asqlitescript using
thecortext.sqlite template. If requested by the
parsing spec, the resultingsqlitescript can be intepreted and thus converted
to a binary database.RequirementsParscival has been set up using PyScaffold, a project generator for
bootstrapping high-quality Python packages. For details and usage information
on PyScaffold seehttps://pyscaffold.org.This project uses PyScaffold in combination with Tox, a generic virtualenv management
and test command line tool acting as frontend to Continuous Integration servers.
A list with all the available tasks is obtained via thetox -avcommand.To prepare your environment you will need to install the following dependencies:pip install -U pip setuptoolspip install -U toxDeploymentvirtualenv .venvsource .venv/bin/activate#...editsetup.cfgtoadddependencies...pip install -e .tox#tocompiledocstox -e docs#tobuilddistributiontox -e buildCreditsParscival is being developed by theCorTexT PlatformandCogniteva SAS.