sqlalchemy: change JSON type to native one.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
jsonalchemy: update `parser.py` for pyparsing 2

* Transform hardcoded decorators into extensions inside the parser
  folder. The new decorators are cataloged as `before`, `on` and `after`
  depending on when they should be evaluated.

* All the decorators that before needed always a tuple now works as
  `@decorator(*args)`.

* Remove `@inherit_from` decorator from field definitions as it is
  replaced by `@extend` and `@override`.

* New decorator extension added `@connect(field_name[, function])` to
  link to fields and propagate changes from one to the other and vice
  versa.

* Remove `include` form field definitions as now registries are used.

* Replace `checker` section by `schema` to perform simple validations,
  like type or length. For more complex check use the checker module.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
jsonalchemy: add `jsonext` as common namespace

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
jsonalchemy: update readers and SmartJson

* Update all parsers to work with the new structutre of the field and
  model definitions, meaning using the extensions properly.

* When translating a new blob there is no need any more to go through
  the `readers` registry, instead call `Reader.translate`. See the
  method docs for more info.

* Update SmartJson to work also with new extensions and decorators
  structure.

* Update JSON for marc producer to adapt it to the new extension
  behavior.

* Add first version of sphinx documentation (to be completed).

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
records: update to new JSONAlchemy

* Add `persistent_identifier` property to the record base object.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
documents: update field and model definitions

Signed-off-by: Esteban J. G. Gabancho <egabancho@gmail.com>
jsonalchemy: fix `six.iteritems` typo

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
documents: fix for model creation

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
jsonalchemy: import and PEP8 fixes

Co-authored-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
jsonalchemy: UUID validation fix

Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
jsonalchemy: `uuid` and `objectid` validator fix

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
base: config PEP8 improvements

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
jsonalchemy: validator test fix

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
jsonalchemy: bug fixes

* Fix record tests when using '001' marc tag.

* Implement `_prepare_blob` method for json_reader to use the model
  field names when using a JSON input.

* Fix model inheritance when overriding fields.

* Fix validation when using models and override fields.

* Enhance `creation_date` field definition.

* PEP8 fixes.

Signed-off-by: Esteban J. G. Gabancho <egabancho@gmail.com>
jsonalchemy: model resolver fix

Signed-off-by: Esteban J. G. Gabancho <egabancho@gmail.com>
bibdocfile: fix for undefined docname in get_text

Signed-off-by: Johnny Mariéthoz <Johnny.Mariethoz@rero.ch>
workflows: add OAIHarvest integration

* Refactors log messages in generic harvesting workflow tasks.

* Adds a new task to convert records with information directly from
  OAI repository models.

* Generic workflow now takes parameter of the given repository.

* Cleans the code of several functions, simplifies the
  harvesting a bit. Changes the way the updating of
  the last_time update.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: refactor holding pen widgets

* Adds mini approval widget.

* Removes every hardcoded url in .js files.

* Adds small helper function in maintable.js

* Adds buttons for multiple record approval.

* Adds new row_formatter to format datatables rows.

* Adds checkbox functionality.

* Fixes breadcrumbs in Holding Pen pages.

* Upgrades templates to bootstrap 3.

* Adds a navbar above Holding Pen table and updates layouts
  across the details page and approval widget page.

* Adds a new tag based UI element for filtering objects in the
  Holding Pen.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: new API in engine and model

* Updates the BibWorkflowObject with new API functions to add
  widgets to trigger an holding pen action.

* Adds a function to BibWorkflowObject to add task results
  using a newly created class WorkflowTaskResult.

* Changes the data preview in Holding Pen back to using
  the previously added get_formatted_data function.

* Widgets and tasks now use these new functions.

* Enriches sample workflow to showcase more tasks.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update workflow tasks

* Adds new filtering task for record categories, allowing to
  specify which categories to trigger an action in holding pen.

* Adds task results to objects in various tasks.

* Updates the sample harvesting workflow to use new bibclassify
  API.

* Several other updates and improvements to workflow tasks.

* Updates the category filtering task to allow regular expression
  syntax when defining which categories to halt, continue or stop
  the workflow.

* Adds new data type attribute to workflows, by Jan Aage Lavik.

* Updates the exception handling to better handle exceptions in
  workflows when running a workflow in a BibSched task.

* Improvements to category filter.

* Adds options for the foreach loop, we can now cache the
  list to go through it will then allow to five the choice
  to the user if he wants to priviledge CPU vs Memory,
  cache uses few CPU but a lot of memory and doesn't provide
  a dynamic list. Without the cache option, we recompute the
  list each time so we use much more CPU but we keep just
  the data needed so few memory, and the list is still
  refresh for each loop.

* Refactors use of extra_data in the engine.

* Changes the task foreach so that obj.data takes the value
  it has before the foreach. In the previous version we were
  keeping the last value of the foreach.

* Adds the function filtering_oai_pmh_identifier which
  allows you to filter the identifier. Mainly used to
  avoid processing already processed one.

* Adds features to  inspire_filter_category, now parameter
  can be passed throught the extra_data of the engine.

* Fixes some bugs with regex in inspire_filter_category

* Adds the function update_last_update which allows you to
  update the last_update time of a repository. It is
  important because the user can choose exactly where in
  the workflows he considers that a crash is not a problem
  for harvested documents.

* Adds the different features needed by the change from
  pickle to message pack in Celery, to the container for
  BibWorkflowObject exchange.

* Adds the inspire_filter_custom task which allows to
  filter record by examinating some field of its bibfield
  It is a generalization of inspire_filter_category.

* Reenables all the functions in full_doc_process
  workflow.

* Adds some control over the number of workflows generated
  it is to avoid the memory saturation or the messaging
  queue limit.

* Adds some protected name for workflows to avoid possible
  conflict with new task an new user data etc.

* Add some functions to control workflows states.

* Fixes PEP8 and clean code

* Deletes non-informative logs.

* Modifies the different workflows to take into account the
  previous modification made.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>